Fix flaky testcase test_reload_configuration_checks so slow device or device without clock chip can pass the test by gechiang · Pull Request #10530 · sonic-net/sonic-mgmt

gechiang · 2023-10-30T20:45:35Z

Description of PR

With some vendors the device may be slow to come up where the database container that needs to be up before this testcase can run may not be ready with just 60 seconds wait time. And this change will allow eliminating false failure for slower devices.
In addition, some device may not have onboard clock chip and relies on the supervisor to provide the current timestamp so the check under common/reboot.py right after reboot occurred the timestamp may go negative. The check for timestamp to determine if reboot took place may end up concluding incorrectly. So this is also addressed in this PR as well.

Summary:
Fixes # (issue) (did not file issue)

Type of change

Bug fix
Testbed and Framework(new/improvement)
Test case(new/improvement)

Back port request

201911
202012
202205
202305

Approach

What is the motivation for this PR?

To reduce flaky test results for some slower platforms and device without onboard clock chip.

How did you verify/test it?

Before the fix:

2023-10-10T22:29:00.6107739Z platform_tests/test_reload_config.py::test_reload_configuration[str3-chassis-sup-2] PASSED [ 16%]
2023-10-10T22:35:01.3980939Z platform_tests/test_reload_config.py::test_reload_configuration_checks[str3-chassis-sup-2] 
2023-10-10T22:35:01.3986805Z -------------------------------- live log call ---------------------------------
2023-10-10T22:35:01.3993036Z 22:35:01 __init__.pytest_runtest_call             L0040 ERROR  | Traceback (most recent call last):
2023-10-10T22:35:01.3997829Z   File "/usr/local/lib/python2.7/dist-packages/_pytest/python.py", line 1464, in runtest
2023-10-10T22:35:01.4001106Z     self.ihook.pytest_pyfunc_call(pyfuncitem=self)
2023-10-10T22:35:01.4004828Z   File "/usr/local/lib/python2.7/dist-packages/pluggy/hooks.py", line 286, in __call__
2023-10-10T22:35:01.4007848Z     return self._hookexec(self, self.get_hookimpls(), kwargs)
2023-10-10T22:35:01.4011322Z   File "/usr/local/lib/python2.7/dist-packages/pluggy/manager.py", line 93, in _hookexec
2023-10-10T22:35:01.4014224Z     return self._inner_hookexec(hook, methods, kwargs)
2023-10-10T22:35:01.4017518Z   File "/usr/local/lib/python2.7/dist-packages/pluggy/manager.py", line 87, in <lambda>
2023-10-10T22:35:01.4020576Z     firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
2023-10-10T22:35:01.4024314Z   File "/usr/local/lib/python2.7/dist-packages/pluggy/callers.py", line 208, in _multicall
2023-10-10T22:35:01.4027056Z     return outcome.get_result()
2023-10-10T22:35:01.4030116Z   File "/usr/local/lib/python2.7/dist-packages/pluggy/callers.py", line 81, in get_result
2023-10-10T22:35:01.4032827Z     _reraise(*ex)  # noqa
2023-10-10T22:35:01.4035829Z   File "/usr/local/lib/python2.7/dist-packages/pluggy/callers.py", line 187, in _multicall
2023-10-10T22:35:01.4038626Z     res = hook_impl.function(*args)
2023-10-10T22:35:01.4041795Z   File "/usr/local/lib/python2.7/dist-packages/_pytest/python.py", line 174, in pytest_pyfunc_call
2023-10-10T22:35:01.4044567Z     testfunction(**testargs)
2023-10-10T22:35:01.4047467Z   File "/azp/_work/71/s/tests/platform_tests/test_reload_config.py", line 100, in test_reload_configuration_checks
2023-10-10T22:35:01.4050597Z     assert "Retry later" in out['stdout']
2023-10-10T22:35:01.4052799Z AssertionError
2023-10-10T22:35:01.4053605Z 
2023-10-10T22:35:01.4246537Z 
2023-10-10T22:35:32.0626752Z platform_tests/test_reload_config.py::test_reload_configuration[str3-chassis-lc4-1] 
2023-10-10T22:35:32.0630645Z -------------------------------- live log call ---------------------------------
2023-10-10T22:35:32.0634215Z 22:35:32 __init__.pytest_runtest_call             L0040 ERROR  | Traceback (most recent call last):
2023-10-10T22:35:32.0638075Z   File "/usr/local/lib/python2.7/dist-packages/_pytest/python.py", line 1464, in runtest
2023-10-10T22:35:32.0640977Z     self.ihook.pytest_pyfunc_call(pyfuncitem=self)
2023-10-10T22:35:32.0644351Z   File "/usr/local/lib/python2.7/dist-packages/pluggy/hooks.py", line 286, in __call__
2023-10-10T22:35:32.0647287Z     return self._hookexec(self, self.get_hookimpls(), kwargs)
2023-10-10T22:35:32.0650674Z   File "/usr/local/lib/python2.7/dist-packages/pluggy/manager.py", line 93, in _hookexec
2023-10-10T22:35:32.0653572Z     return self._inner_hookexec(hook, methods, kwargs)
2023-10-10T22:35:32.0656856Z   File "/usr/local/lib/python2.7/dist-packages/pluggy/manager.py", line 87, in <lambda>
2023-10-10T22:35:32.0659856Z     firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
2023-10-10T22:35:32.0664011Z   File "/usr/local/lib/python2.7/dist-packages/pluggy/callers.py", line 208, in _multicall
2023-10-10T22:35:32.0666790Z     return outcome.get_result()
2023-10-10T22:35:32.0669829Z   File "/usr/local/lib/python2.7/dist-packages/pluggy/callers.py", line 81, in get_result
2023-10-10T22:35:32.0672507Z     _reraise(*ex)  # noqa
2023-10-10T22:35:32.0675545Z   File "/usr/local/lib/python2.7/dist-packages/pluggy/callers.py", line 187, in _multicall
2023-10-10T22:35:32.0678308Z     res = hook_impl.function(*args)
2023-10-10T22:35:32.0681439Z   File "/usr/local/lib/python2.7/dist-packages/_pytest/python.py", line 174, in pytest_pyfunc_call
2023-10-10T22:35:32.0684242Z     testfunction(**testargs)
2023-10-10T22:35:32.0687047Z   File "/azp/_work/71/s/tests/platform_tests/test_reload_config.py", line 34, in test_reload_configuration
2023-10-10T22:35:32.0689946Z     if config_force_option_supported(duthost):
2023-10-10T22:35:32.0692817Z   File "/azp/_work/71/s/tests/common/config_reload.py", line 62, in config_force_option_supported
2023-10-10T22:35:32.0697069Z     out = duthost.shell("config reload -h", executable="/bin/bash")
2023-10-10T22:35:32.0700093Z   File "/azp/_work/71/s/tests/common/devices/multi_asic.py", line 112, in _run_on_asics
2023-10-10T22:35:32.0703452Z     return getattr(self.sonichost, self.multi_asic_attr)(*module_args, **complex_args)
2023-10-10T22:35:32.0706027Z   File "/azp/_work/71/s/tests/common/devices/base.py", line 78, in _run
2023-10-10T22:35:32.0707257Z     res = self.module(*module_args, **complex_args)[self.hostname]
2023-10-10T22:35:32.0708773Z   File "/usr/local/lib/python2.7/dist-packages/pytest_ansible/module_dispatcher/v28.py", line 159, in _run
2023-10-10T22:35:32.0710253Z     raise AnsibleConnectionFailure("Host unreachable", dark=cb.unreachable, contacted=cb.contacted)
2023-10-10T22:35:32.0711509Z AnsibleConnectionFailure: Host unreachable
2023-10-10T22:35:32.0711922Z 
2023-10-10T22:35:32.1282754Z 
2023-10-10T22:36:02.7471051Z platform_tests/test_reload_config.py::test_reload_configuration_checks[str3-chassis-lc4-1] 
2023-10-10T22:36:02.7475702Z -------------------------------- live log call ---------------------------------
2023-10-10T22:36:02.7479244Z 22:36:02 __init__.pytest_runtest_call             L0040 ERROR  | Traceback (most recent call last):
2023-10-10T22:36:02.7483316Z   File "/usr/local/lib/python2.7/dist-packages/_pytest/python.py", line 1464, in runtest
2023-10-10T22:36:02.7486766Z     self.ihook.pytest_pyfunc_call(pyfuncitem=self)
2023-10-10T22:36:02.7490163Z   File "/usr/local/lib/python2.7/dist-packages/pluggy/hooks.py", line 286, in __call__
2023-10-10T22:36:02.7493070Z     return self._hookexec(self, self.get_hookimpls(), kwargs)
2023-10-10T22:36:02.7498372Z   File "/usr/local/lib/python2.7/dist-packages/pluggy/manager.py", line 93, in _hookexec
2023-10-10T22:36:02.7505491Z     return self._inner_hookexec(hook, methods, kwargs)
2023-10-10T22:36:02.7512412Z   File "/usr/local/lib/python2.7/dist-packages/pluggy/manager.py", line 87, in <lambda>
2023-10-10T22:36:02.7520158Z     firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
2023-10-10T22:36:02.7528235Z   File "/usr/local/lib/python2.7/dist-packages/pluggy/callers.py", line 208, in _multicall
2023-10-10T22:36:02.7534224Z     return outcome.get_result()
2023-10-10T22:36:02.7541824Z   File "/usr/local/lib/python2.7/dist-packages/pluggy/callers.py", line 81, in get_result
2023-10-10T22:36:02.7547448Z     _reraise(*ex)  # noqa
2023-10-10T22:36:02.7554555Z   File "/usr/local/lib/python2.7/dist-packages/pluggy/callers.py", line 187, in _multicall
2023-10-10T22:36:02.7560620Z     res = hook_impl.function(*args)
2023-10-10T22:36:02.7567750Z   File "/usr/local/lib/python2.7/dist-packages/_pytest/python.py", line 174, in pytest_pyfunc_call
2023-10-10T22:36:02.7608582Z     testfunction(**testargs)
2023-10-10T22:36:02.7611725Z   File "/azp/_work/71/s/tests/platform_tests/test_reload_config.py", line 86, in test_reload_configuration_checks
2023-10-10T22:36:02.7613567Z     if not config_force_option_supported(duthost):
2023-10-10T22:36:02.7615429Z   File "/azp/_work/71/s/tests/common/config_reload.py", line 62, in config_force_option_supported
2023-10-10T22:36:02.7617475Z     out = duthost.shell("config reload -h", executable="/bin/bash")
2023-10-10T22:36:02.7619308Z   File "/azp/_work/71/s/tests/common/devices/multi_asic.py", line 112, in _run_on_asics
2023-10-10T22:36:02.7622020Z     return getattr(self.sonichost, self.multi_asic_attr)(*module_args, **complex_args)
2023-10-10T22:36:02.7623823Z   File "/azp/_work/71/s/tests/common/devices/base.py", line 78, in _run
2023-10-10T22:36:02.7625521Z     res = self.module(*module_args, **complex_args)[self.hostname]
2023-10-10T22:36:02.7627548Z   File "/usr/local/lib/python2.7/dist-packages/pytest_ansible/module_dispatcher/v28.py", line 159, in _run
2023-10-10T22:36:02.7628921Z     raise AnsibleConnectionFailure("Host unreachable", dark=cb.unreachable, contacted=cb.contacted)
2023-10-10T22:36:02.7630092Z AnsibleConnectionFailure: Host unreachable
2023-10-10T22:36:02.7630937Z 
2023-10-10T22:36:02.8098861Z 
2023-10-10T22:43:41.3783751Z platform_tests/test_reload_config.py::test_reload_configuration[str3-chassis-lc3-1] PASSED [ 83%]
2023-10-10T22:57:53.4785470Z platform_tests/test_reload_config.py::test_reload_configuration_checks[str3-chassis-lc3-1] PASSED [100%]

After the fix:

platform_tests/test_reload_config.py::test_reload_configuration[str3-chassis-sup-2] PASSED [ 16%]
platform_tests/test_reload_config.py::test_reload_configuration_checks[str3-chassis-sup-2] PASSED [ 33%]
platform_tests/test_reload_config.py::test_reload_configuration[str3-chassis-lc4-1] PASSED [ 50%]
platform_tests/test_reload_config.py::test_reload_configuration_checks[str3-chassis-lc4-1] PASSED [ 66%]
platform_tests/test_reload_config.py::test_reload_configuration[str3-chassis-lc2-1] PASSED [ 83%]
platform_tests/test_reload_config.py::test_reload_configuration_checks[str3-chassis-lc2-1] PASSED [100%]

------ generated xml file: /var/xxx/sonic-mgmt-int/tests/logs/tr.xml ------
========================= 6 passed in 4449.18 seconds ==========================

… allowed for database conainer to come up a little slower

mssonicbld · 2023-10-30T20:46:39Z

The pre-commit check detected issues in the files touched by this pull request.
The pre-commit check is a mandatory check, please fix detected issues.

Detailed pre-commit check results:
trim trailing whitespace.................................................Failed - hook id: trailing-whitespace - exit code: 1 - files were modified by this hook Fixing tests/platform_tests/test_reload_config.py fix end of files.........................................................Passed check yaml...........................................(no files to check)Skipped check for added large files..............................................Passed check python ast.........................................................Passed flake8...................................................................Passed flake8...............................................(no files to check)Skipped check conditional mark sort..........................(no files to check)Skipped

To run the pre-commit checks locally, you can follow below steps:

Ensure that default python is python3. In sonic-mgmt docker container, default python is python2. You can run
the check by activating the python3 virtual environment in sonic-mgmt docker container or outside of sonic-mgmt
docker container.
Ensure that the pre-commit package is installed:

sudo pip install pre-commit

Go to repository root folder
Install the pre-commit hooks:

pre-commit install

Use pre-commit to check staged file:

pre-commit

Alternatively, you can check committed files using:

pre-commit run --from-ref <commit_id> --to-ref <commit_id>

mssonicbld · 2023-10-30T20:51:50Z

The pre-commit check detected issues in the files touched by this pull request.
The pre-commit check is a mandatory check, please fix detected issues.

Detailed pre-commit check results:
trim trailing whitespace.................................................Failed - hook id: trailing-whitespace - exit code: 1 - files were modified by this hook Fixing tests/platform_tests/test_reload_config.py fix end of files.........................................................Passed check yaml...........................................(no files to check)Skipped check for added large files..............................................Passed check python ast.........................................................Passed flake8...................................................................Passed flake8...............................................(no files to check)Skipped check conditional mark sort..........................(no files to check)Skipped

To run the pre-commit checks locally, you can follow below steps:

Ensure that default python is python3. In sonic-mgmt docker container, default python is python2. You can run
the check by activating the python3 virtual environment in sonic-mgmt docker container or outside of sonic-mgmt
docker container.
Ensure that the pre-commit package is installed:

sudo pip install pre-commit

Go to repository root folder
Install the pre-commit hooks:

pre-commit install

Use pre-commit to check staged file:

pre-commit

Alternatively, you can check committed files using:

pre-commit run --from-ref <commit_id> --to-ref <commit_id>

…ast commit

gechiang · 2023-10-30T21:04:10Z

@vperumal Please check on this PR as why without this fix it is not failing for you?

gechiang · 2023-11-07T16:38:41Z

@abdosi , @wangxin can you help review/approve?
Thanks!

SuvarnaMeenakshi · 2023-11-27T19:34:58Z

tests/common/reboot.py

+    # some device does not have onchip clock and requires obtaining system time a little later from ntp
+    # or SUP to obtain the correct time so if the uptime is less than original device time, it means it
+    # is most likely due to this issue which we can wait a little more until the correct time is set in place.
+    if float(dut_uptime.strftime("%s")) < float(dut_datetime.strftime("%s")):


Will it be always be the case that the time drifted backwards and not ahead?

exactly! that is why this change is needed. in most platforms that has on board clock chip, there should be time difference that is "positive". For platform that does not have clock chip and rely on time sync and "slow" we will always hit this check and causes the testcase to failed prematurely...

… device without clock chip can pass the test (sonic-net#10530) * Fix flaky testcase test_reload_configuration_checks so slow device is allowed for database conainer to come up a little slower

mssonicbld · 2023-12-14T08:33:59Z

Cherry-pick PR to 202305: #11016

… device without clock chip can pass the test (sonic-net#10530) * Fix flaky testcase test_reload_configuration_checks so slow device is allowed for database conainer to come up a little slower

mssonicbld · 2023-12-14T08:34:07Z

Cherry-pick PR to 202205: #11017

… device without clock chip can pass the test (#10530) * Fix flaky testcase test_reload_configuration_checks so slow device is allowed for database conainer to come up a little slower

Fix flaky testcase test_reload_configuration_checks so slow device is…

9da4233

… allowed for database conainer to come up a little slower

gechiang requested a review from prgeor as a code owner October 30, 2023 20:45

gechiang requested review from SuvarnaMeenakshi and abdosi October 30, 2023 20:45

Removed trailing spaces from platform/test_reload_config.py from my l…

d4b77f3

…ast commit

vperumal approved these changes Oct 31, 2023

View reviewed changes

gechiang added chassis Request for 202205 branch Request for 202305 branch labels Nov 1, 2023

gechiang self-assigned this Nov 1, 2023

SuvarnaMeenakshi reviewed Nov 27, 2023

View reviewed changes

SuvarnaMeenakshi approved these changes Nov 27, 2023

View reviewed changes

gechiang merged commit 08e5b77 into sonic-net:master Nov 27, 2023

kevinskwang added Approved for 202205 branch Approved for 202305 branch labels Dec 14, 2023

mssonicbld added the Created PR to 202305 branch label Dec 14, 2023

mssonicbld mentioned this pull request Dec 14, 2023

[action] [PR:10530] Fix flaky testcase test_reload_configuration_checks so slow device or device without clock chip can pass the test #11016

Merged

7 tasks

mssonicbld added the Created PR to 202205 branch label Dec 14, 2023

mssonicbld mentioned this pull request Dec 14, 2023

[action] [PR:10530] Fix flaky testcase test_reload_configuration_checks so slow device or device without clock chip can pass the test #11017

Merged

7 tasks

mssonicbld added the Included in 202205 branch label Dec 14, 2023

mssonicbld added Included in 202305 branch and removed Created PR to 202305 branch Created PR to 202205 branch labels Dec 14, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix flaky testcase test_reload_configuration_checks so slow device or device without clock chip can pass the test#10530

Fix flaky testcase test_reload_configuration_checks so slow device or device without clock chip can pass the test#10530
gechiang merged 2 commits intosonic-net:masterfrom
gechiang:master

gechiang commented Oct 30, 2023

Uh oh!

mssonicbld commented Oct 30, 2023

Uh oh!

mssonicbld commented Oct 30, 2023

Uh oh!

gechiang commented Oct 30, 2023

Uh oh!

gechiang commented Nov 7, 2023

Uh oh!

SuvarnaMeenakshi Nov 27, 2023

Uh oh!

gechiang Nov 27, 2023

Uh oh!

mssonicbld commented Dec 14, 2023

Uh oh!

mssonicbld commented Dec 14, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

gechiang commented Oct 30, 2023

Description of PR

Type of change

Back port request

Approach

What is the motivation for this PR?

How did you verify/test it?

Uh oh!

mssonicbld commented Oct 30, 2023

Uh oh!

mssonicbld commented Oct 30, 2023

Uh oh!

gechiang commented Oct 30, 2023

Uh oh!

gechiang commented Nov 7, 2023

Uh oh!

SuvarnaMeenakshi Nov 27, 2023

Choose a reason for hiding this comment

Uh oh!

gechiang Nov 27, 2023

Choose a reason for hiding this comment

Uh oh!

mssonicbld commented Dec 14, 2023

Uh oh!

mssonicbld commented Dec 14, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants