Add an option in run_tests.sh to run tests in the input order and make kvm tests running in input order#2419
Conversation
tests/run_tests.sh
Outdated
| fi | ||
| # Ignore the scripts specified in $SKIP_SCRIPTS | ||
| TEST_CASES=$(python -c "print '\n'.join(set('''$all_scripts'''.split()) - set('''$SKIP_SCRIPTS'''.split()))" | sort) | ||
| TEST_CASES=$(python -c "print '\n'.join([testcase for testcase in list('''$all_scripts'''.split()) if testcase not in set('''$SKIP_SCRIPTS'''.split())])") |
There was a problem hiding this comment.
there are two mode, group mode and individual mode, I do not know if we need to run alphabetic order for the group mode, and only preserve the order for the individual mode.
There was a problem hiding this comment.
Made running in input order as a command-line option.
|
Normally, all test cases should be independent to each other. So running order should make no difference to the result. But there are some cases that may left routes/IP on DUT or change the configuration of DUT, which may fail following cases. In such case, running them in alphabetic order enables us to analyze it more easily. Maybe we could add another command line option to toggle the dafault |
|
agree with you that test cases should be independent to each other. but sometimes, we do want to make sure certain test cases can be run first for examle warm reboot so that we can test behavior after the warm reboot. but I think adding a command line option is probably a better design. |
shi-su
changed the title
5 participants
Description of PR
Summary: Add an option in run_tests.sh to run tests in the input order and make kvm tests running in input order
Fixes # (issue)
Type of change
Approach
What is the motivation for this PR?
Currently, run_tests.sh sort tests in alphabetical order. Running in the input order may provide more freedom in organizing test cases. Adding running in input order as an option in run_tests.sh.
How did you do it?
Add an option in run_tests.sh to run tests in the input order rather than alphabetical order. And make kvm tests running in input order.
How did you verify/test it?
Verified by running tests locally.
Any platform specific information?
Supported testbed topology if it's a new test case?
Documentation