Enhance qos tests to support single-asic, multi-asic, and multi-dut testing

[vmittal-msft]

Description of PR

Summary:
Fixes # (issue)

This is same as PR #6946 from 'master' branch that can't be cherry-picked without merge conflicts into '202205' branch.

The existing QoS (test_qos_sai.py) is written to accomodata a single asic on a single Dut. But, we require the same tests to be executed against a T2 chassis (with single/multi-asic linecards) and multi-asic pizza boxes.

Type of change

• Bug fix
• Testbed and Framework(new/improvement)
• Test case(new/improvement)

Back port request

• 201911
• 202012
• 202205

Approach

What is the motivation for this PR?

All the test cases create a list of src and dst ports. For the different modes, here is the distribution of the src and dst ports:

• single_asic: The src and dst ports are on the same asic on the same linecard.
• single_dut_multi_asic: On a multi-asic DUT/linecard, the src port is on an asic, while the dst ports are on another asic on the same DUT/linecard
• multi_dut: The src port is on an asic on one of the DUT/linecards, and the dst port is on another asic on another DUT/linecard. This is currently only required for T2 topology

How did you do it?

Approach to accomplish this is the following:

• All the tests have to parameterized for the 3 modes defined above.

  • This is done using the 'select_src_and_dst_dut_and_asic' fixture that is parameterized for 'single_asic', 'single_dut_multi_asic', 'multi_dut' - Based on the mode, it sets the src_dut_index, dst_dut_index, src_asic_index and dst_asic_index
  • Added fixture 'get_src_dst_asic_and_duts' that returns dictionary of the src_dut_index, dst_dut_index, src_asic_index, and dst_asic_index, and the src_dut and dst_dut (instances of MultiAsicSonicHost), src_asic and dst_asic (instances of Asic), and also a list of all DUTs and all Asics

• dutConfig is modified such that testPortIds and testPortIps are collecting from all the duts and asics involved and stored in a dictionary with key being the dutIndex and value being a dictionary per asic index.

  • __buildTestPorts then sets the src and dst ports based on the src_dut_index, dst_dut_index, src_asic_index and dst_asic_index

• All the other fixtures and tests, we use 'get_src_dst_asci_and_duts' fixture instead of enum_rand_one_frontend_hostname and enum_frontend_index.

  • The code instead the fixtures and tests is modified to the actions on the correct src/dst dut or asic. For example: - swap_syncd fixture would swap syncd docker on all DUT's (both src and dst) instead of just one DUT as before. - stopServices - do it all_duts (src and dst duts)

• Similarly, changes to saitests involved dealing with multiple DUTs (and thus multiple sai clients) and modifying other data structure like 'interface_to_front_mapping' in sai_base_test.py and port_list, sai_port_list, front_port_list in switch.py to deal with multiple duts (modified to be dictionary with key being 'src' and 'dst')

  • tests in sai_qos_tests.py pass src_dut_index, src_asic_index, dst_dut_index and dst_asic_index in the testParams.
    • The saitests classes then use this to do the actions on the right client and ports.

Assumptions:

• For multi-dut, we are assuming that hwsku for all the cards are same.

How did you verify/test it?

Verified these changes on SONIC T0/T1/T2 topologies for different vendors HWSKUs

Any platform specific information?

Supported testbed topology if it's a new test case?

Documentation

[vmittal-msft]

#7556 PR was reverted since few failures were seen. Opening this new PR to handle those issues and push the changes into 202205 branch.

[judyjoseph]

The sonic-mgmt tests is skipped in PR tester, can you trigger it again
Also add the test run for topologies if you have results saved somewhere.

[vmittal-msft]

/azp run

[azure-pipelines]

Azure Pipelines could not run because the pipeline triggers exclude this branch/path.

[arlakshm]

#7556 PR was reverted since few failures were seen. Opening this new PR to handle those issues and push the changes into 202205 branch.

@vmittal-msft, can you highlight what were the failures seen because of this change and what changes were added to this PR to fixes those

[arlakshm]

changes look lgtm

[vmittal-msft]

@arlakshm please check the commit history. We have mainly added fixes for Mellanox as well as Cisco platforms. We used to have failures for those two SKUs. Those are passing now.