How to use in non-manual mode?

[bitphage]

Hi. Just trying to play with k8s & SR-IOV CNI. I compiled plugins and put binaries to /opt/cni/bin.

I have SR-IOV configured on the host:

[root@srv-neon ~]# ip li sh dev enp5s0f0
3: enp5s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether d4:85:64:4c:80:8c brd ff:ff:ff:ff:ff:ff
    vf 0 MAC 02:7c:32:ed:63:27, spoof checking on, link-state auto
    vf 1 MAC 66:d8:02:77:aa:aa, spoof checking on, link-state auto
    vf 2 MAC be:e8:1d:fa:0a:b3, spoof checking on, link-state auto
    vf 3 MAC 02:21:41:90:12:7d, spoof checking on, link-state auto

My k8s setup consist of master node and one worker node.

Then I created cni config like this (only on the worker node):

{
    "name": "mynet",
    "type": "sriov",
    "master": "enp5s0f0",
    "vlan": "28",
    "ipam": {
        "type": "host-local",
        "subnet": "192.168.91.0/24",
        "routes": [
            { "dst": "0.0.0.0/0" }
        ],
        "gateway": "192.168.91.1"
    }
}

Then enabled cni for kubelet (worker node):

KUBELET_ARGS="--cgroup-driver=systemd --network-plugin=cni"

Then created simple pod (on the master node)

# cat kube-nginx2.yml 
apiVersion: v1
kind: Pod
metadata:
  name: www2
spec:
  containers:
    - name: nginx
      image: nginx

# kubectl create -f kube-nginx2.yml

Then I tried to check network availability inside CT (kubectl - master, nsenter - worker) with:

kubectl get po <POD-NAME> -o jsonpath='{.status.containerStatuses[0].containerID}' | cut -c 10-21
PID=$(docker inspect --format '{{ .State.Pid }}' <ID>)
nsenter -t ${PID} -n ip addr
nsenter -t ${PID} -n ip link
nsenter -t ${PID} -n ping 192.168.91.1

I found I cannot ping 192.168.91.1 from inside CT because no VLAN tag actually was assigned to the VF which was passed to the CT. I assume this is a bug. ?

I checked traffic originating from the pod via tcpdump on the host machine and found no VLAN tag in the packets.

192.168.91.1/24 is assigned to a linux router sitting on VLAN 28.

Next, I'm deleting pod (master)

kubectl delete pod www2

And getting errors (kubelet log):

Sep 19 17:43:26 srv-neon.tgt72.ru kubelet[22822]: E0919 17:43:26.444681   22822 cni.go:278] Error deleting network: failed to open netns %!!(MISSING)q(<nil>): failed to Statfs "": no such file or directory
Sep 19 17:43:26 srv-neon.tgt72.ru kubelet[22822]: E0919 17:43:26.446795   22822 remote_runtime.go:109] StopPodSandbox "f18b8bf5512b1b05ea39039a598cae2b5c55df233e63a1047e89c1b455b6ca55" from runtime service failed: rpc error: code = 2 desc = NetworkPlugin cni failed to teardown pod "www2_default" network: failed to open netns %!!(MISSING)q(<nil>): failed to Statfs "": no such file or directory
Sep 19 17:43:26 srv-neon.tgt72.ru kubelet[22822]: E0919 17:43:26.446835   22822 kuberuntime_gc.go:138] Failed to stop sandbox "f18b8bf5512b1b05ea39039a598cae2b5c55df233e63a1047e89c1b455b6ca55" before removing: rpc error: code = 2 desc = NetworkPlugin cni failed to teardown pod "www2_default" network: failed to open netns %!!(MISSING)q(<nil>): failed to Statfs "": no such file or directory
Sep 19 17:43:26 srv-neon.tgt72.ru kubelet[22822]: W0919 17:43:26.451390   22822 docker_sandbox.go:263] NetworkPlugin cni failed on the status hook for pod "www_default": Cannot find the network namespace, skipping pod network status for container {"docker" "1e68c0d580a0707c4f91ce7f98a3a35e31d22d3ca741a520209ebadb8ed23302"}

I'm new to k8s, so I can miss something, please give me some ideas what I missing. I would like to debug this situation to get SR-IOV cni plugin working.

So the things I'm missing:

• is it supposed to work with k8s in non-manual mode?
• VLAN is not getting assigned - is this a bug?

[rkamudhan]

Can you list the virtual functions in /sys/class/net/enp5s0f0/device/ ?

[bitphage]

Do you mean this?

# cat /sys/class/net/enp5s0f0/device/sriov_numvfs 
4

Or:

[root@srv-neon ~]# ls -l /sys/class/net/enp5s0f0/device/ |grep virt
lrwxrwxrwx. 1 root root      0 Sep 19 18:04 iommu -> ../../../virtual/iommu/dmar0
lrwxrwxrwx. 1 root root      0 Sep 10 14:33 virtfn0 -> ../0000:05:10.0
lrwxrwxrwx. 1 root root      0 Sep 10 14:19 virtfn1 -> ../0000:05:10.2
lrwxrwxrwx. 1 root root      0 Sep 19 18:04 virtfn2 -> ../0000:05:10.4
lrwxrwxrwx. 1 root root      0 Sep 19 18:04 virtfn3 -> ../0000:05:10.6

[bitphage]

On the host machine I'm able to assign a VLAN to one of the VFs and ping router:

ip link add link dev11 name dev11.28 type vlan id 28
ip a a 192.168.91.100/24 dev dev11.28
ip li set dev11.28 up
ping 192.168.91.1
PING 192.168.91.1 (192.168.91.1) 56(84) bytes of data.
64 bytes from 192.168.91.1: icmp_seq=1 ttl=64 time=0.402 ms
64 bytes from 192.168.91.1: icmp_seq=2 ttl=64 time=0.303 ms

dev11 is a vf1:

# ip a sh dev11
11: dev11: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 66:d8:02:77:aa:aa brd ff:ff:ff:ff:ff:ff
    inet6 fe80::b5c0:dd75:8f3f:1d4b/64 scope link 
       valid_lft forever preferred_lft forever

vf 1 MAC 66:d8:02:77:aa:aa, spoof checking on, link-state auto

[hustcat]

@bitfag We should pass vlan with CNI_ARGS, not network config file. Because different VF can have different VLAN ID. Please see Extra arguments

[bitphage]

@hustcat ok. Maybe clarify this in README?

[rkamudhan]

@hustcat @bitfag I will update the README.md..

How to turn vf on in centos7