Skip to content

[util tests] add markers to help reduce util group execution logs #1903

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
yxieca merged 2 commits into from
Jul 15, 2020
Merged

[util tests] add markers to help reduce util group execution logs #1903

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
30df733
[util tests] add markers to help reduce util group execution logs
yxieca Jul 14, 2020
c35a6d1
set return value according to the feature test cases run
yxieca Jul 14, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion tests/pytest.ini
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,12 +1,13 @@
[pytest]
addopts = -ra --module-path=../ansible/library/ --show-capture=no --ignore=ptftests --ignore=acstests --ignore=saitests --ignore=test_vrf.py
markers:
acl: ACL tests
bsl: BSL tests
reboot: tests which perform SONiC reboot
port_toggle: tests which toggle ports
disable_loganalyzer: make to disable automatic loganalyzer
broadcom: test specific to Broadcom platform
pretest: tests are run before feature/regression test cases to prepare the DUT/environment.
posttest: tests are run after feature/regression test cases to cleanup the DUT/environment and/or collect diagnostics.
Copy link
Contributor

@neethajohn neethajohn Jul 14, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Which test uses this marker? is it a placeholder?

Copy link
Collaborator Author

@yxieca yxieca Jul 14, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Place holder for now. the cleanup tests will go to this group.

sanity_check: override the default sanity check settings
topology: specify which topology testcase can be executed on: (t0, t1, ptf, etc)
platform: specify which platform testcase can be executed on: (physical, virtual, broadcom, mellanox, etc)
Expand Down
37 changes: 25 additions & 12 deletions tests/run_tests.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -95,12 +95,14 @@ function setup_test_options()
fi

if [[ x"${OMIT_FILE_LOG}" == x"True" ]]; then
UTIL_LOGGING_OPTIONS=""
PRET_LOGGING_OPTIONS=""
Copy link
Contributor

@tahmed-dev tahmed-dev Jul 15, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PRET?

POST_LOGGING_OPTIONS=""
TEST_LOGGING_OPTIONS=""
else
mkdir -p ${LOG_PATH}

UTIL_LOGGING_OPTIONS="--junit-xml=${LOG_PATH}/util.xml --log-file=${LOG_PATH}/util.log"
PRET_LOGGING_OPTIONS="--junit-xml=${LOG_PATH}/pretest.xml --log-file=${LOG_PATH}/pretest.log"
POST_LOGGING_OPTIONS="--junit-xml=${LOG_PATH}/posttest.xml --log-file=${LOG_PATH}/posttest.log"
TEST_LOGGING_OPTIONS="--junit-xml=${LOG_PATH}/tr.xml --log-file=${LOG_PATH}/test.log"
fi
UTIL_TOPOLOGY_OPTIONS="--topology util"
Expand Down Expand Up @@ -132,7 +134,8 @@ function run_debug_tests()
echo "TESTBED_FILE: ${TESTBED_FILE}"
echo "TEST_LOGGING_OPTIONS: ${TEST_LOGGING_OPTIONS}"
echo "TEST_TOPOLOGY_OPTIONS: ${TEST_TOPOLOGY_OPTIONS}"
echo "UTIL_LOGGING_OPTIONS: ${UTIL_LOGGING_OPTIONS}"
echo "PRET_LOGGING_OPTIONS: ${PRET_LOGGING_OPTIONS}"
echo "POST_LOGGING_OPTIONS: ${POST_LOGGING_OPTIONS}"
echo "UTIL_TOPOLOGY_OPTIONS: ${UTIL_TOPOLOGY_OPTIONS}"

echo "PYTEST_COMMON_OPTS: ${PYTEST_COMMON_OPTS}"
Expand All @@ -141,27 +144,26 @@ function run_debug_tests()
function prepare_dut()
{
echo "=== Preparing DUT for subsequent tests ==="
py.test ${PYTEST_COMMON_OPTS} ${UTIL_LOGGING_OPTIONS} ${UTIL_TOPOLOGY_OPTIONS} ${EXTRA_PARAMETERS}
py.test ${PYTEST_COMMON_OPTS} ${PRET_LOGGING_OPTIONS} ${UTIL_TOPOLOGY_OPTIONS} ${EXTRA_PARAMETERS} -m pretest

# Give some delay for the newly announced routes to propagate.
sleep 120
}

function cleanup_dut()
{
echo "=== Cleaning up DUT after tests ==="
py.test ${PYTEST_COMMON_OPTS} ${POST_LOGGING_OPTIONS} ${UTIL_TOPOLOGY_OPTIONS} ${EXTRA_PARAMETERS} -m posttest
}

function run_group_tests()
{
if [[ x"${BYPASS_UTIL}" == x"False" ]]; then
prepare_dut
fi
echo "=== Running tests in groups ==="
py.test ${PYTEST_COMMON_OPTS} ${TEST_LOGGING_OPTIONS} ${TEST_TOPOLOGY_OPTIONS} ${EXTRA_PARAMETERS} ${TEST_CASES}
}

function run_individual_tests()
{
if [[ x"${BYPASS_UTIL}" == x"False" ]]; then
prepare_dut
fi

SKIP_SCRIPTS="${SKIP_SCRIPTS} test_announce_routes.py test_nbr_health.py"

ignores=$(python -c "print '|'.join('''$SKIP_FOLDERS'''.split())")
Expand Down Expand Up @@ -258,4 +260,15 @@ done
validate_parameters
setup_test_options

run_${TEST_METHOD}_tests
if [[ x"${BYPASS_UTIL}" == x"False" ]]; then
prepare_dut
fi

RC=0
run_${TEST_METHOD}_tests || RC=$?

if [[ x"${BYPASS_UTIL}" == x"False" ]]; then
cleanup_dut
fi

exit ${RC}
1 change: 1 addition & 0 deletions tests/test_announce_routes.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
import pytest

pytestmark = [
pytest.mark.pretest,
pytest.mark.topology('util') #special marker
]

Expand Down
1 change: 1 addition & 0 deletions tests/test_nbr_health.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
pytestmark = [
pytest.mark.sanity_check(skip_sanity=True),
pytest.mark.disable_loganalyzer,
pytest.mark.pretest,
pytest.mark.topology('util') #special marker
]

Expand Down