Support for not using all ports on hardware DUT for testing

[sanmalho-git]

Description of PR

Summary:
Support for not using all ports on hardware DUT for testing and thus not connecting all the ports to a fanout switch.

Type of change

• Bug fix
• Testbed and Framework(new/improvement)
• Test case(new/improvement)

Approach

What is the motivation for this PR?

Currently, it is required that all ports on DUT are in use and are connected to a fanout.
However, there is a need to be able to run tests where all ports are not in use. Specifically, when dealing with

• new hardware boxes
• boxes with a front panel port used for in-band management
• boxes with lots of ports and multiple asics, were every port on every asic is not required to be covered
• chassis as a DUT, where the number of ports can be in hundreds
• higher bandwidth ports like 400G
  • hard to go from 400G down to 1/10G

Also, majority of the basic functional testing can be done without testing all the ports on the DUT.

How did you do it?

To support above in orchestration, following changes were made:

• dut_fp_ports is changed to be a dictionary instead of a list, and
• vlan_base as an optional ansible variable passed into testbed-cli.sh.

In current orchestration in add-topo, the dut_fp_ports is a list of nic’s on the testbed server corresponding to the ports on the DUT. In the topology file, the 'host_interfaces' and 'vlans' for VM's are defined as an offset from the 'vlan_base'. When we 'bind' the topology (using vm_topology module), it uses this offset as the index into the dut_fp_ports to get the corresponding nic on the testbed server to the DUT's port. So, when having lesser number of elements in dut_fp_ports than the offsets defined in the topology results in index of out bound exception.

By changing dut_fp_ports to be a dictionary with key being this offset from 'vlan_base' and the value being the actual nic on the testbed server corresponding to the DUT's port, we avoid the above issue. This 'vlan_base' defaults to 0 (to give backward compatability) and can be specified as an extra ansible arg (using -e option). If it is not specified (has value 0), then the key is the same as the index in the original dut_fp_ports list.

For example, let's consider that we have a topology with only 2 ports (3 and 51) on the DUT connected to the fanout, with fanout vlans 103 and 151 respectively,
'vlan_base' of 100, and the trunk port on the testbed server being 'eno2'. Then dut_fp_ports would look like:

        {
          '3' :  'eno2.103',
          `51' : 'eno2.151
        }

For a topology where we 32 ports connected to fanout with vlans 100 - 132 on trunk port eno2, and vlan_base is not specified, then the resulting dut_fp_ports would be

     {
       '0':  'eno2.100',
       '1':  'eno2.101',
       '2':  'eno2.102',
        .
        .
        '31': 'eno2.132'
     }

How did you verify/test it?

• Ran add/remove-topo against KVM SONiC DUT and ceos containers against t0 topology and validated BGP and ptf connectivity

    ./testbed-cli.sh -t vtestbed.csv -m veos.vtb -k ceos add-topo vms-kvm-t0 password.txt 
    ./testbed-cli.sh -t vtestbed.csv -m veos.vtb deploy-mg vms-kvm-t0 lab password.txt
    ansible-playbook -i lab -l vlab-01 test_sonic.yml -e testbed_name=vms-kvm-t0 -e testcase_name=fdb -e testbed_file=vtestbed.csv 
    ./testbed-cli.sh -t vtestbed.csv -m veos.vtb -k ceos remove-topo vms-kvm-t0 password.txt

• Ran add/remove-topo against a SONiC DUT with all ports connected to a fanout against t0 topology and validated BGP and ptf connectivity

    ./testbed-cli.sh -t testbed.csv -m veos.vtb -k ceos add-topo vms-s6100-t0 password.txt
    ./testbed-cli.sh -t testbed.csv -m veos.vtb deploy-mg vms-s6100-t0 lab password.txt
    ./testbed-cli.sh -t testbed.csv -m veos.vtb -k ceos remove-topo vms-s6100-t0 password.txt

• Ran add/remove-topo against a SONiC DUT with only 2 ports connected to fanout against
  topology defining only 1 VM and 1 'host-interfaces' and validated BGP and ptf connectivity

    ./testbed-cli.sh -t testbed.csv -m veos.vtb -k ceos add-topo vms-s6100-2p password.txt -e vlan_base=100
    ./testbed-cli.sh -t testbed.csv -m veos.vtb deploy-mg vms-s6100-2p lab password.txt
    ./testbed-cli.sh -t testbed.csv -m veos.vtb -k ceos remove-topo vms-s6100-2p password.txt -e vlan_base=100

Any platform specific information?

Supported testbed topology if it's a new test case?

Documentation

[ghost]

All CLA requirements met.

[yxieca]

Approved with 2 very cosmetic issues.

[sanmalho-git]

The solution proposed works if we have a one-to-one mapping between a leaf fanout switch and DUT. But, if we want to share the leaf fanout amongst multiple DUT's, then this restricts the ports on the DUT that are connected to the fanout - they have to be all unique and contiguous amongst the multiple DUTs.

Working on an implementation the would use the connection graph to create the dut_fp_ports dictionary. The connection graph 'device_port_vlans' has the DUT port and fanout vlan info. For example:
"device_port_vlans": {
"Ethernet19": {
"mode": "Access",
"vlanids": "114",
"vlanlist": [
114
]
},
"Ethernet9": {
"mode": "Access",
"vlanids": "113",
"vlanlist": [
113
]
}
},

This along with the device hwsku can help get the right key mapping for the dut_fp_ports dictionary.

[wangxin]

retest vsimage please

[jleveque]

@sanmalho-git: The newly-added platform API tests (specifically the SFP tests) run under the assumption that transceivers are connected to all ports on the device. With this change, we will also need to update those tests to understand which ports should have transceivers connected. Could you please also provide an example of what a test should do to determine whether a port is expected to be connected?

[sanmalho-git]

@jleveque: We only list the ports that are connected in the connection graph (eg. lab_connection_graph.xml). In our test, we use 'conn_graphs' ansible module to get these connections and expect transciever to be present only on those ports that are in the connection graph.

eg. sonic_lab_links.csv:

et_6448m_52x-r0_pizza3,,,Ethernet3,ixr7220D1pizza3,,,Ethernet3,1000,103,Access
et_6448m_52x-r0_pizza3,,,Ethernet51,ixr7220D1pizza3,,,Ethernet51,100000,151,Access

One other challenge for us right now is that we have SONiC box with 1G copper ports as well. We are trying to figure out how we would be able to distinguish those as ports present in connection graph, but still no transceiver is present on them (eg. Ethernet3 from above sample sonic_lab_links.csv). Currently, we are using the 'bandwidth' to determine this. If it is 1000, then we expect no transceiver. Our SONiC box has 48 copper 1G port, and 4 10G SFP ports. But, this won't work for 1G Fiber ports.

[jleveque]

One other challenge for us right now is that we have SONiC box with 1G copper ports as well. We are trying to figure out how we would be able to distinguish those as ports present in connection graph, but still no transceiver is present on them (eg. Ethernet3 from above sample sonic_lab_links.csv). Currently, we are using the 'bandwidth' to determine this. If it is 1000, then we expect no transceiver. Our SONiC box has 48 copper 1G port, and 4 10G SFP ports. But, this won't work for 1G Fiber ports.

In this case, we may want to consider adding a "transceiver type" and/or "cable type" to the connection graph. It would also be beneficial for my SFP tests, in order to know which type of transceiver to expect when testing.

[yxieca]

@sanmalho-git can you address the merge conflict?

[sanmalho-git]

This approach doesn't work well for us, and so are closing this pull request. Will open another PR with a better solution.