failed to move vf 0 to netns

[dlohin]

I have been fighting an issue with trying to get Kubernetes to schedule pods with Multus and SR-IOV for at least two days. We are building a pod on every host that has three interfaces one going to Calico, one SR-IOV on one NIC port and one going to SR-IOV NIC port on an XL710. I tried this with the Intel-corp sr-iov repo and this one.

When I try to deploy the pod I get the following error message: Failed to move vf 0 to netns: file exists

More information:

Apr 24 09:59:06 localhost kubelet: E0424 09:59:06.386309 2754 cni.go:259] Error adding network: netplugin failed but error parsing its diagnostic message "{\n "ips": [\n {\n "version": "4",\n "address": "10.244.192.209/32"\n }\n ],\n "dns": {}\n}{\n "code": 100,\n "msg": "Multus: error in invoke Delegate add - \"sriov\": failed to set up pod interface \"net1\" from the device \"enp216s0f2\": failed to move vf 0 to netns: file exists"\n}": invalid character '{' after top-level value
Apr 24 09:59:06 localhost kubelet: E0424 09:59:06.386352 2754 cni.go:227] Error while adding to cni network: netplugin failed but error parsing its diagnostic message "{\n "ips": [\n {\n "version": "4",\n "address": "10.244.192.209/32"\n }\n ],\n "dns": {}\n}{\n "code": 100,\n "msg": "Multus: error in invoke Delegate add - \"sriov\": failed to set up pod interface \"net1\" from the device \"enp216s0f2\": failed to move vf 0 to netns: file exists"\n}": invalid character '{' after top-level value

My interfaces look like this:

[root@minion-19425 ~]# ./listvf.sh
Virtual Functions on Intel Corporation Ethernet Controller XL710 for 40GbE QSFP+. (enp216s0f1):
PCI BDF		Interface
=======		=========
0000:d8:06.0	eth0
0000:d8:06.1	eth3

Virtual Functions on Intel Corporation Ethernet Controller XL710 for 40GbE QSFP+. (enp216s0f2):
PCI BDF		Interface
=======		=========
0000:d8:0a.0	eth1
0000:d8:0a.1	eth4

Virtual Functions on Intel Corporation Ethernet Controller XL710 for 40GbE QSFP+. (enp216s0f3):
PCI BDF		Interface
=======		=========
0000:d8:0e.0	eth2
0000:d8:0e.1	eth5

The complaint is always about the first VF on the enp216s0f1 interface on all systems. I found that if I do the following:
ip netns add test
ip link set eth0 netns test

Which basically adds that nic to another namespace and therefore makes it "schedulable" it works fine.

When I run listvf.sh after it has been scheduled:

[root@minion-26997 ~]# ./listvf.sh
Virtual Functions on Intel Corporation Ethernet Controller XL710 for 40GbE QSFP+. (enp216s0f1):
PCI BDF		Interface
=======		=========
0000:d8:06.0
0000:d8:06.1

Virtual Functions on Intel Corporation Ethernet Controller XL710 for 40GbE QSFP+. (enp216s0f2):
PCI BDF		Interface
=======		=========
0000:d8:0a.0
0000:d8:0a.1	eth4

Virtual Functions on Intel Corporation Ethernet Controller XL710 for 40GbE QSFP+. (enp216s0f3):
PCI BDF		Interface
=======		=========
0000:d8:0e.0	eth2
0000:d8:0e.1	eth5

From that output you can see that both VF 0 and VF1 are now consumed because I had to "burn" vf0.

This could very well be something I am doing wrong. I have had this working fine in the past. Some questions:

1. Do you know what file it is talking about? I am assuming it is something in /proc/
2. How can I reset these interfaces properly so that these go away?
3. I have the i40evf driver loaded, that needs to be there, is that correct?

Thanks for your help!

[hustcat]

It seem that 'net1' interface exist in pod container

[dlohin]

Is there a way to figure out which container? I have done it with no pods scheduled to those hosts at all and am still getting the same results. When we set these up I have seen the listvf command switch the name from net0 to net1 and then it gets "stuck" on the host. From what I have seen it appears to create the network interface on the host and then the network interface disappears from the host once it goes into the namespace. Perhaps the issue is with multus? Could it be that it is creating both interfaces with the same names when it should be using net0, net1, etc?