|
| 1 | +# Pytest organization proposal |
| 2 | + |
| 3 | +This proposal intends to achieve the following |
| 4 | + - Have a standard way of categorizing tests |
| 5 | + - Have some guidelines around test file organization |
| 6 | + - Have a master wrapper for test execution |
| 7 | + - Follow common documentation style |
| 8 | + - Test result collection |
| 9 | + |
| 10 | +## Test categorization |
| 11 | +Leverage pytest custom markers to group tests based on topology, asic, features, device type and connection type. |
| 12 | +Every testcase needs to have a topology marker. Feature markers are recommended for any feature test that are getting added. |
| 13 | +'Device_type' is optional but needs to be specified if there is a specific requirement that the test needs a physical DUT as opposed to a VS. The same criteria applies for 'connection_type' |
| 14 | + |
| 15 | +``` |
| 16 | +pytest.ini |
| 17 | +[pytest] |
| 18 | +markers: |
| 19 | + topology(topo_name): The topologies this particular testcase can run against. topo_name can be individual topology names like 't0', 't1', 'ptf', 'any' or a comma separated like ('t0', 't1') if supported on multiple topologies |
| 20 | + asic(vendor_name): used for asic specific test(broadcom, mellanox etc) |
| 21 | + feature(feature_name): feature this test is written for. eg. acl, nat |
| 22 | + connection_type(name): names can be 'fabric' (which indicates the presence of a fanout switch) or 'direct' if a testcase uses directly connected links |
| 23 | + device_type(name): name can 'physical' (if this test requires a physical dut) or 'vs' (if this test can be run on a virtual switch) |
| 24 | +
|
| 25 | +``` |
| 26 | +conftest.py |
| 27 | + |
| 28 | +``` |
| 29 | +def pytest_addoption(parser): |
| 30 | + parser.addoption("--topology", action="store", metavar="NAME", |
| 31 | + help="only run tests matching the topology NAME") |
| 32 | +
|
| 33 | +def pytest_runtest_setup(item): |
| 34 | + toponames = [mark.args for mark in item.iter_markers(name="topology")] |
| 35 | + if toponames: |
| 36 | + cfg_topos = item.config.getoption("--topology").split(',') |
| 37 | + if all(topo not in toponames[0] for topo in cfg_topos): |
| 38 | + pytest.skip("test requires topology in {!r}".format(toponames)) |
| 39 | + else: |
| 40 | + if item.config.getoption("--topology"): |
| 41 | + pytest.skip("test does not match topology") |
| 42 | +
|
| 43 | +``` |
| 44 | + |
| 45 | +Sample test file: test_topo.py |
| 46 | + |
| 47 | +``` |
| 48 | +@pytest.mark.topology('t0', 't1') |
| 49 | +def test_all(): |
| 50 | + assert 1 == 1 |
| 51 | +
|
| 52 | +@pytest.mark.topology('t0') |
| 53 | +def test_t0(): |
| 54 | + assert 1 == 1 |
| 55 | +
|
| 56 | +
|
| 57 | +@pytest.mark.topology('any') |
| 58 | +def test_any(): |
| 59 | + assert 1 == 1 |
| 60 | +
|
| 61 | +``` |
| 62 | + |
| 63 | +Sample test file: test_notopo.py |
| 64 | + |
| 65 | +``` |
| 66 | +def test_notopo(): |
| 67 | + assert 1 == 1 |
| 68 | +
|
| 69 | +``` |
| 70 | + |
| 71 | +Test run |
| 72 | + |
| 73 | +``` |
| 74 | +py.test --inventory inv --host-pattern dut1 --module-path ../ansible/library/ --testbed tb --testbed_file tb.csv --topology t1 test_topo.py test_notopo.py -rA |
| 75 | +
|
| 76 | +platform linux2 -- Python 2.7.12, pytest-4.6.9, py-1.8.1, pluggy-0.13.1 |
| 77 | +ansible: 2.8.7 |
| 78 | +rootdir: /var/nejo/Networking-acs-sonic-mgmt/tests, inifile: pytest.ini |
| 79 | +plugins: ansible-2.2.2 |
| 80 | +collected 4 items |
| 81 | +
|
| 82 | +test_topo.py::test_all PASSED [ 25%] |
| 83 | +test_topo.py::test_t0 PASSED [ 50%] |
| 84 | +test_topo.py::test_any SKIPPED [ 75%] |
| 85 | +test_notopo.py::test_notopo SKIPPED [100%] |
| 86 | +
|
| 87 | +.... |
| 88 | +
|
| 89 | +.... |
| 90 | +PASSED test_topo.py::test_all |
| 91 | +SKIPPED [1] /var/nejo/Networking-acs-sonic-mgmt/tests/conftest.py:293: test requires topology in [('t0',)] |
| 92 | +SKIPPED [1] /var/nejo/Networking-acs-sonic-mgmt/tests/conftest.py:293: test requires topology in [('any',)] |
| 93 | +SKIPPED [1] /var/nejo/Networking-acs-sonic-mgmt/tests/conftest.py:295: test does not match topology |
| 94 | +
|
| 95 | +``` |
| 96 | + |
| 97 | +## Test file organization |
| 98 | +- Have 2 broad categories (platform and feature). Feature specific tests and their helpers go into specific feature folders. |
| 99 | + |
| 100 | +``` |
| 101 | +tests |
| 102 | + |_ common |
| 103 | + |_ platform |
| 104 | + |_ ptftests |
| 105 | + |_ nat |
| 106 | + |_ test_nat_bindings.py |
| 107 | + |_ files |
| 108 | + |_ all helpers for the nat feature |
| 109 | + |_ acl |
| 110 | +
|
| 111 | +``` |
| 112 | + |
| 113 | +- Any reusable code needs to go under tests/common |
| 114 | + |
| 115 | +- File naming convention |
| 116 | + The objective here is to provide meaningful names for helper files/testcase files so that the user gets a general idea of the file contents. |
| 117 | + |
| 118 | + |
| 119 | +## Master wrapper |
| 120 | +Make it easier to run a nightly test against a feature/platform/topology from the command line. Have something similar to the 'ansible/testbed-cli.sh' script which can be invoked with just the basic parameters (testbed name, what flavor of test to run) |
| 121 | + |
| 122 | + |
| 123 | +## Documentation style |
| 124 | +Follow a common style of documentation for test methods which can be used by some tool to generate html content |
| 125 | + |
| 126 | + |
| 127 | +## Test result collection |
| 128 | +Use the --junitxml attribute to collect test results. Can leverage the existing format used in sonic-utilities/sonic-swss repo for reporting test results. |
0 commit comments