[parametize by ports] add infrastructure to allow test parametized by ports

[yxieca]

Summary:
Fixes # (issue)

Type of change

• Bug fix
• Testbed and Framework(new/improvement)
• Test case(new/improvement)

Approach

What is the motivation for this PR?

We want to reduce big loop test cases that loop through for example all ports. test_link_flap.py is one example.
We would like to break these test cases down to individual testcases for each port. so that we can exactly know which port had issue.
With this change, test_link_flap is ready for multi-dut tests.

How did you do it?

Creating infrastructure to loop through all port or port channels, or the subset of oper_up and/or admin_up.

Assumptions:

• Run test_pretest before actual test.
• test_update_testbed_metadata will populate testbed information
  and save to tests/metadata/<testbed_name>.json.

Available parameterizes:

• all_ports : all ports available on target testbed.
• oper_up_ports : all ports in oper_up state.
• admin_up_ports : all ports in admin_up state.
• all_pcs : all port channel available on target testbed.
• oper_up_pcs : all port channel in oper_up state.
• admin_up_pcs : all port channel in admin_up state.

Parameter format:
"<dut_hostname>|<port_name>" e.g. str-dx010-acs-1|Ethernet8
Please use the provided helper function decode_dut_port_name() to decode into dutname and port name. (see sample code in test_link_flap)

Expectations:

• when metadata file is not found, test will still be called with a single parameter 'unknown|unknown'.
• when 'unknown|unknown' is passed in as parameter, test should warn and loop through all ports (port channels).

Signed-off-by: Ying Xie [email protected]

How did you verify/test it?

1. test without metadata:
   ================================================================================== test session starts ===================================================================================
   platform linux2 -- Python 2.7.12, pytest-4.6.5, py-1.9.0, pluggy-0.13.1
   ansible: 2.8.12
   rootdir: /var/src/sonic-mgmt/tests, inifile: pytest.ini
   plugins: ansible-2.2.2, xdist-1.28.0, metadata-1.10.0, forked-1.3.0, html-1.22.1, repeat-0.8.0
   collected 1 item

platform_tests/test_link_flap.py::test_link_flap[unknown|unknown]
------------------------------------------------------------------------------------- live log call --------------------------------------------------------------------------------------
06:36:55 WARNING test_link_flap.py:__build_test_candidates:74: Failed to get ports enumerated as parameter. Fall back to test all ports
PASSED [100%]

==================================================================================== warnings summary ====================================================================================

2. Generating the metadata:
   ================================================================================== test session starts ===================================================================================
   platform linux2 -- Python 2.7.12, pytest-4.6.5, py-1.9.0, pluggy-0.13.1
   ansible: 2.8.12
   rootdir: /var/src/sonic-mgmt/tests, inifile: pytest.ini
   plugins: ansible-2.2.2, xdist-1.28.0, metadata-1.10.0, forked-1.3.0, html-1.22.1, repeat-0.8.0
   collected 1 item

test_pretest.py::test_update_testbed_metadata PASSED [100%]

------------------------------------------------------------------------ generated xml file: /tmp/logs-01/tr.xml -------------------------------------------------------------------------
=============================================================================== 1 passed in 75.95 seconds ================================================================================

3. enumerate the ports:
   platform linux2 -- Python 2.7.12, pytest-4.6.5, py-1.9.0, pluggy-0.13.1
   ansible: 2.8.12
   rootdir: /var/src/sonic-mgmt/tests, inifile: pytest.ini
   plugins: ansible-2.2.2, xdist-1.28.0, metadata-1.10.0, forked-1.3.0, html-1.22.1, repeat-0.8.0
   collected 32 items

platform_tests/test_link_flap.py::test_link_flap[str-dx010-acs-1|Ethernet8] PASSED [ 3%]
platform_tests/test_link_flap.py::test_link_flap[str-dx010-acs-1|Ethernet0] PASSED [ 6%]
platform_tests/test_link_flap.py::test_link_flap[str-dx010-acs-1|Ethernet4] PASSED [ 9%]
platform_tests/test_link_flap.py::test_link_flap[str-dx010-acs-1|Ethernet108] PASSED [ 12%]
platform_tests/test_link_flap.py::test_link_flap[str-dx010-acs-1|Ethernet100] PASSED [ 15%]
platform_tests/test_link_flap.py::test_link_flap[str-dx010-acs-1|Ethernet104] PASSED [ 18%]
platform_tests/test_link_flap.py::test_link_flap[str-dx010-acs-1|Ethernet68] PASSED [ 21%]
... ...

[lgtm-com]

This pull request introduces 1 alert and fixes 1 when merging e9f996a into 4703756 - view on LGTM.com

new alerts:

• 1 for Unused import

fixed alerts:

• 1 for Unused import

[lgtm-com]

This pull request fixes 1 alert when merging 4bcdff3 into 553e9ff - view on LGTM.com

fixed alerts:

• 1 for Unused import

[lgtm-com]

This pull request fixes 1 alert when merging 7591239 into 553e9ff - view on LGTM.com

fixed alerts:

• 1 for Unused import

[lgtm-com]

This pull request fixes 1 alert when merging d391ed2 into 553e9ff - view on LGTM.com

fixed alerts:

• 1 for Unused import