[test runner] add a script to run pytests#1861
Conversation
Provide common infrastructure to run pytest. Test can be run in
two different methods:
1. Group execution: test case are executed in groups. First execute
the util group, then executed the specified topology group.
1.1. in group execution mode, user can specify option -c to run
a (list of) test cases/files.
2. Individual execution: first execute the util group. Then execute
specified topology group individually.
Signed-off-by: Ying Xie <[email protected]>
daall
left a comment
daall
left a comment
There was a problem hiding this comment.
Added a few random thoughts, I don't think any of them are blocking issues though.
tests/run_tests.sh
Outdated
| TEST_CASES="" | ||
| OMIT_FILE_LOG="False" | ||
| BYPASS_UTIL="False" | ||
| RETAIN_SUCCUSESS_LOG="False" |
There was a problem hiding this comment.
| RETAIN_SUCCUSESS_LOG="False" | |
| RETAIN_SUCCESSES_LOG="False" |
typo
There was a problem hiding this comment.
My face red. Thanks!
daall
left a comment
daall
left a comment
There was a problem hiding this comment.
Just one more q, but overall LGTM.
tests/run_tests.sh
Outdated
| function run_group_tests() | ||
| { | ||
| if [[ x"${BYPASS_UTIL}" == x"False" ]]; then | ||
| echo "=== Runing utility test cases ===" |
There was a problem hiding this comment.
Could this language be confusing? Should we describe this as "pre-steps" or "test setup" or something similar? I'm just wondering because we're not really testing anything per se (at least I don't think so?), we're preparing the testbed for further steps.
tests/run_tests.sh
Outdated
| { | ||
| if [[ x"${BYPASS_UTIL}" == x"False" ]]; then | ||
| echo "=== Runing pre-test cases ===" | ||
| py.test ${PYTEST_COMMON_OPTS} ${UTIL_LOGGING_OPTIONS} ${UTIL_TOPOLOGY_OPTIONS} -k "not test_restart_syncd" ${EXTRA_PARAMETERS} |
There was a problem hiding this comment.
This test_restart_syncd case in tests/platform_tests/test_sequential_restart.py has been marked as skip. It's unnecessary to skip it again here.
There was a problem hiding this comment.
It would be better to add sleep 120 here to wait a while for the DUT to learn all the IP routes.
There was a problem hiding this comment.
Thanks! Changed both.
tests/run_tests.sh
Outdated
| py.test ${PYTEST_COMMON_OPTS} ${UTIL_LOGGING_OPTIONS} ${UTIL_TOPOLOGY_OPTIONS} -k "not test_restart_syncd" ${EXTRA_PARAMETERS} | ||
| fi | ||
| echo "=== Running tests in groups ===" | ||
| py.test ${PYTEST_COMMON_OPTS} ${TEST_LOGGING_OPTIONS} ${TEST_TOPOLOGY_OPTIONS} -k "not test_restart_syncd" ${EXTRA_PARAMETERS} ${TEST_CASES} |
There was a problem hiding this comment.
Same for test_restart_syncd here.
tests/run_tests.sh
Outdated
| { | ||
| if [[ x"${BYPASS_UTIL}" == x"False" ]]; then | ||
| echo "=== Runing pre-test cases ===" | ||
| py.test ${PYTEST_COMMON_OPTS} ${UTIL_LOGGING_OPTIONS} ${UTIL_TOPOLOGY_OPTIONS} -k "not test_restart_syncd" ${EXTRA_PARAMETERS} |
There was a problem hiding this comment.
Same for test_restart_syncd here.
8768089 (HEAD -> 202012, origin/202012) Remove exec from platform_reboot_plugin call to handle any hang issue. (sonic-net#1879) ae5d90c Validate input of ```config mirror_session add``` (sonic-net#1825) 44d3a3b [show][config] fix the muxcable commands for interface naming mode (sonic-net#1862) 0a4933e [TH3] Skipp Control Plane Assist on WARM Reboot for TH3 HWSKUs (sonic-net#1861) Signed-off-by: vaibhav-dahiya <[email protected]>
3 participants
Summary:
Fixes # (issue)
Type of change
Approach
What is the motivation for this PR?
Provide a common way for the community to run pytests.
How did you do it?
Provide common infrastructure to run pytest. Test can be run in
2 different methods:
the util group, then executed the specified topology group.
1.1. in group execution mode, user can specify option -c to run
a (list of) test cases/files.
specified topology group individually.
Signed-off-by: Ying Xie [email protected]
How did you verify/test it?
yinxi@acs-trusty8:/var/src/sonic-mgmt/tests$ ./run_tests.sh -d str-dx010-acs-1 -n vms3-t1-dx010-1 -t t1,any -s "test_vrf.py test_vrf_attr.py test_mgmtvrf.py copp/test_copp.py" -i /var/src/sonic-mgmt/ansible/str,/var/src/sonic-mgmt/ansible/veo -e --collect-only -m group -o -u -c platform_tests/test_reboot.py
=== running tests in groups ===
================================================================================== test session starts ===================================================================================
platform linux2 -- Python 2.7.12, pytest-4.6.5, py-1.8.1, pluggy-0.13.1
ansible: 2.8.7
rootdir: /var/src/sonic-mgmt/tests, inifile: pytest.ini
plugins: ansible-2.2.2, xdist-1.28.0, forked-1.1.3, repeat-0.8.0
collected 7 items
<Package /var/src/sonic-mgmt/tests/platform_tests>
<Function test_power_off_reboot[15]>
<Function test_power_off_reboot[5]>