Enhance qos tests to support single-asic, multi-asic, and multi-dut testing

[vmittal-msft]

Description of PR

Summary:
Fixes # (issue)

This is same as PR #6946 from 'master' branch that can't be cherry-picked without merge conflicts into '202205' branch.

The existing QoS (test_qos_sai.py) is written to accomodata a single asic on a single Dut. But, we require the same tests to be executed against a T2 chassis (with single/multi-asic linecards) and multi-asic pizza boxes.

Type of change

• Bug fix
• Testbed and Framework(new/improvement)
• Test case(new/improvement)

Back port request

• 201911
• 202012
• 202205

Approach

What is the motivation for this PR?

All the test cases create a list of src and dst ports. For the different modes, here is the distribution of the src and dst ports:

• single_asic: The src and dst ports are on the same asic on the same linecard.
• single_dut_multi_asic: On a multi-asic DUT/linecard, the src port is on an asic, while the dst ports are on another asic on the same DUT/linecard
• multi_dut: The src port is on an asic on one of the DUT/linecards, and the dst port is on another asic on another DUT/linecard. This is currently only required for T2 topology

How did you do it?

Approach to accomplish this is the following:

• All the tests have to parameterized for the 3 modes defined above.

  • This is done using the 'select_src_and_dst_dut_and_asic' fixture that is parameterized for 'single_asic', 'single_dut_multi_asic', 'multi_dut' - Based on the mode, it sets the src_dut_index, dst_dut_index, src_asic_index and dst_asic_index
  • Added fixture 'get_src_dst_asic_and_duts' that returns dictionary of the src_dut_index, dst_dut_index, src_asic_index, and dst_asic_index, and the src_dut and dst_dut (instances of MultiAsicSonicHost), src_asic and dst_asic (instances of Asic), and also a list of all DUTs and all Asics

• dutConfig is modified such that testPortIds and testPortIps are collecting from all the duts and asics involved and stored in a dictionary with key being the dutIndex and value being a dictionary per asic index.

  • __buildTestPorts then sets the src and dst ports based on the src_dut_index, dst_dut_index, src_asic_index and dst_asic_index

• All the other fixtures and tests, we use 'get_src_dst_asci_and_duts' fixture instead of enum_rand_one_frontend_hostname and enum_frontend_index.

  • The code instead the fixtures and tests is modified to the actions on the correct src/dst dut or asic. For example: - swap_syncd fixture would swap syncd docker on all DUT's (both src and dst) instead of just one DUT as before. - stopServices - do it all_duts (src and dst duts)

• Similarly, changes to saitests involved dealing with multiple DUTs (and thus multiple sai clients) and modifying other data structure like 'interface_to_front_mapping' in sai_base_test.py and port_list, sai_port_list, front_port_list in switch.py to deal with multiple duts (modified to be dictionary with key being 'src' and 'dst')

  • tests in sai_qos_tests.py pass src_dut_index, src_asic_index, dst_dut_index and dst_asic_index in the testParams.
    • The saitests classes then use this to do the actions on the right client and ports.

Assumptions:

• For multi-dut, we are assuming that hwsku for all the cards are same.

How did you verify/test it?

Verified these changes on SONIC T0/T1/T2 topologies for different vendors HWSKUs

Any platform specific information?

Supported testbed topology if it's a new test case?

Documentation

[linux-foundation-easycla]

The committers listed above are authorized under a signed CLA.

• ✅ login: vmittal-msft (0fcc6a4395628bd909a05436c8ce460fa747bd59, 29315f749ef60707fb21170d593fd6039b4e01ae, e316e3baf3f16b0c3ab3cff5b7c2dff201a9a458)

[vmittal-msft]

#8059 PR was reverted since few failures were seen. Opening this new PR to handle those issues and push the changes into 202205 branch.

[vmittal-msft]

master PR : #8149

[jarias-lfx]

/easycla

[judyjoseph]

@vmittal-msft Can you add the tests/pipeline details also if it is available,

[vmittal-msft]

I have run T0/T1/T2 nightly pipelines on different HWSKUs and they seem to work fine.