[action] [PR:10530] Fix flaky testcase test_reload_configuration_checks so slow device or device without clock chip can pass the test

[mssonicbld]

Description of PR

With some vendors the device may be slow to come up where the database container that needs to be up before this testcase can run may not be ready with just 60 seconds wait time. And this change will allow eliminating false failure for slower devices.
In addition, some device may not have onboard clock chip and relies on the supervisor to provide the current timestamp so the check under common/reboot.py right after reboot occurred the timestamp may go negative. The check for timestamp to determine if reboot took place may end up concluding incorrectly. So this is also addressed in this PR as well.

Summary:
Fixes # (issue) (did not file issue)

Type of change

• Bug fix
• Testbed and Framework(new/improvement)
• Test case(new/improvement)

Back port request

• 201911
• 202012
• 202205
• 202305

Approach

What is the motivation for this PR?

To reduce flaky test results for some slower platforms and device without onboard clock chip.

How did you verify/test it?

Before the fix:

2023-10-10T22:29:00.6107739Z platform_tests/test_reload_config.py::test_reload_configuration[str3-chassis-sup-2] PASSED [ 16%]
2023-10-10T22:35:01.3980939Z platform_tests/test_reload_config.py::test_reload_configuration_checks[str3-chassis-sup-2] 
2023-10-10T22:35:01.3986805Z -------------------------------- live log call ---------------------------------
2023-10-10T22:35:01.3993036Z 22:35:01 __init__.pytest_runtest_call L0040 ERROR | Traceback (most recent call last):
2023-10-10T22:35:01.3997829Z File "/usr/local/lib/python2.7/dist-packages/_pytest/python.py", line 1464, in runtest
2023-10-10T22:35:01.4001106Z self.ihook.pytest_pyfunc_call(pyfuncitem=self)
2023-10-10T22:35:01.4004828Z File "/usr/local/lib/python2.7/dist-packages/pluggy/hooks.py", line 286, in __call__
2023-10-10T22:35:01.4007848Z return self._hookexec(self, self.get_hookimpls(), kwargs)
2023-10-10T22:35:01.4011322Z File "/usr/local/lib/python2.7/dist-packages/pluggy/manager.py", line 93, in _hookexec
2023-10-10T22:35:01.4014224Z return self._inner_hookexec(hook, methods, kwargs)
2023-10-10T22:35:01.4017518Z File "/usr/local/lib/python2.7/dist-packages/pluggy/manager.py", line 87, in <lambda>
2023-10-10T22:35:01.4020576Z firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
2023-10-10T22:35:01.4024314Z File "/usr/local/lib/python2.7/dist-packages/pluggy/callers.py", line 208, in _multicall
2023-10-10T22:35:01.4027056Z return outcome.get_result()
2023-10-10T22:35:01.4030116Z File "/usr/local/lib/python2.7/dist-packages/pluggy/callers.py", line 81, in get_result
2023-10-10T22:35:01.4032827Z _reraise(*ex) # noqa
2023-10-10T22:35:01.4035829Z File "/usr/local/lib/python2.7/dist-packages/pluggy/callers.py", line 187, in _multicall
2023-10-10T22:35:01.4038626Z res = hook_impl.function(*args)
2023-10-10T22:35:01.4041795Z File "/usr/local/lib/python2.7/dist-packages/_pytest/python.py", line 174, in pytest_pyfunc_call
2023-10-10T22:35:01.4044567Z testfunction(**testargs)
2023-10-10T22:35:01.4047467Z File "/azp/_work/71/s/tests/platform_tests/test_reload_config.py", line 100, in test_reload_configuration_checks
2023-10-10T22:35:01.4050597Z assert "Retry later" in out['stdout']
2023-10-10T22:35:01.4052799Z AssertionError
2023-10-10T22:35:01.4053605Z 
2023-10-10T22:35:01.4246537Z 
2023-10-10T22:35:32.0626752Z platform_tests/test_reload_config.py::test_reload_configuration[str3-chassis-lc4-1] 
2023-10-10T22:35:32.0630645Z -------------------------------- live log call ---------------------------------
2023-10-10T22:35:32.0634215Z 22:35:32 __init__.pytest_runtest_call L0040 ERROR | Traceback (most recent call last):
2023-10-10T22:35:32.0638075Z File "/usr/local/lib/python2.7/dist-packages/_pytest/python.py", line 1464, in runtest
2023-10-10T22:35:32.0640977Z self.ihook.pytest_pyfunc_call(pyfuncitem=self)
2023-10-10T22:35:32.0644351Z File "/usr/local/lib/python2.7/dist-packages/pluggy/hooks.py", line 286, in __call__
2023-10-10T22:35:32.0647287Z return self._hookexec(self, self.get_hookimpls(), kwargs)
2023-10-10T22:35:32.0650674Z File "/usr/local/lib/python2.7/dist-packages/pluggy/manager.py", line 93, in _hookexec
2023-10-10T22:35:32.0653572Z return self._inner_hookexec(hook, methods, kwargs)
2023-10-10T22:35:32.0656856Z File "/usr/local/lib/python2.7/dist-packages/pluggy/manager.py", line 87, in <lambda>
2023-10-10T22:35:32.0659856Z firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
2023-10-10T22:35:32.0664011Z File "/usr/local/lib/python2.7/dist-packages/pluggy/callers.py", line 208, in _multicall
2023-10-10T22:35:32.0666790Z return outcome.get_result()
2023-10-10T22:35:32.0669829Z File "/usr/local/lib/python2.7/dist-packages/pluggy/callers.py", line 81, in get_result
2023-10-10T22:35:32.0672507Z _reraise(*ex) # noqa
2023-10-10T22:35:32.0675545Z File "/usr/local/lib/python2.7/dist-packages/pluggy/callers.py", line 187, in _multicall
2023-10-10T22:35:32.0678308Z res = hook_impl.function(*args)
2023-10-10T22:35:32.0681439Z File "/usr/local/lib/python2.7/dist-packages/_pytest/python.py", line 174, in pytest_pyfunc_call
2023-10-10T22:35:32.0684242Z testfunction(**testargs)
2023-10-10T22:35:32.0687047Z File "/azp/_work/71/s/tests/platform_tests/test_reload_config.py", line 34, in test_reload_configuration
2023-10-10T22:35:32.0689946Z if config_force_option_supported(duthost):
2023-10-10T22:35:32.0692817Z File "/azp/_work/71/s/tests/common/config_reload.py", line 62, in config_force_option_supported
2023-10-10T22:35:32.0697069Z out = duthost.shell("config reload -h", executable="/bin/bash")
2023-10-10T22:35:32.0700093Z File "/azp/_work/71/s/tests/common/devices/multi_asic.py", line 112, in _run_on_asics
2023-10-10T22:35:32.0703452Z return getattr(self.sonichost, self.multi_asic_attr)(*module_args, **complex_args)
2023-10-10T22:35:32.0706027Z File "/azp/_work/71/s/tests/common/devices/base.py", line 78, in _run
2023-10-10T22:35:32.0707257Z res = self.module(*module_args, **complex_args)[self.hostname]
2023-10-10T22:35:32.0708773Z File "/usr/local/lib/python2.7/dist-packages/pytest_ansible/module_dispatcher/v28.py", line 159, in _run
2023-10-10T22:35:32.0710253Z raise AnsibleConnectionFailure("Host unreachable", dark=cb.unreachable, contacted=cb.contacted)
2023-10-10T22:35:32.0711509Z AnsibleConnectionFailure: Host unreachable
2023-10-10T22:35:32.0711922Z 
2023-10-10T22:35:32.1282754Z 
2023-10-10T22:36:02.7471051Z platform_tests/test_reload_config.py::test_reload_configuration_checks[str3-chassis-lc4-1] 
2023-10-10T22:36:02.7475702Z -------------------------------- live log call ---------------------------------
2023-10-10T22:36:02.7479244Z 22:36:02 __init__.pytest_runtest_call L0040 ERROR | Traceback (most recent call last):
2023-10-10T22:36:02.7483316Z File "/usr/local/lib/python2.7/dist-packages/_pytest/python.py", line 1464, in runtest
2023-10-10T22:36:02.7486766Z self.ihook.pytest_pyfunc_call(pyfuncitem=self)
2023-10-10T22:36:02.7490163Z File "/usr/local/lib/python2.7/dist-packages/pluggy/hooks.py", line 286, in __call__
2023-10-10T22:36:02.7493070Z return self._hookexec(self, self.get_hookimpls(), kwargs)
2023-10-10T22:36:02.7498372Z File "/usr/local/lib/python2.7/dist-packages/pluggy/manager.py", line 93, in _hookexec
2023-10-10T22:36:02.7505491Z return self._inner_hookexec(hook, methods, kwargs)
2023-10-10T22:36:02.7512412Z File "/usr/local/lib/python2.7/dist-packages/pluggy/manager.py", line 87, in <lambda>
2023-10-10T22:36:02.7520158Z firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
2023-10-10T22:36:02.7528235Z File "/usr/local/lib/python2.7/dist-packages/pluggy/callers.py", line 208, in _multicall
2023-10-10T22:36:02.7534224Z return outcome.get_result()
2023-10-10T22:36:02.7541824Z File "/usr/local/lib/python2.7/dist-packages/pluggy/callers.py", line 81, in get_result
2023-10-10T22:36:02.7547448Z _reraise(*ex) # noqa
2023-10-10T22:36:02.7554555Z File "/usr/local/lib/python2.7/dist-packages/pluggy/callers.py", line 187, in _multicall
2023-10-10T22:36:02.7560620Z res = hook_impl.function(*args)
2023-10-10T22:36:02.7567750Z File "/usr/local/lib/python2.7/dist-packages/_pytest/python.py", line 174, in pytest_pyfunc_call
2023-10-10T22:36:02.7608582Z testfunction(**testargs)
2023-10-10T22:36:02.7611725Z File "/azp/_work/71/s/tests/platform_tests/test_reload_config.py", line 86, in test_reload_configuration_checks
2023-10-10T22:36:02.7613567Z if not config_force_option_supported(duthost):
2023-10-10T22:36:02.7615429Z File "/azp/_work/71/s/tests/common/config_reload.py", line 62, in config_force_option_supported
2023-10-10T22:36:02.7617475Z out = duthost.shell("config reload -h", executable="/bin/bash")
2023-10-10T22:36:02.7619308Z File "/azp/_work/71/s/tests/common/devices/multi_asic.py", line 112, in _run_on_asics
2023-10-10T22:36:02.7622020Z return getattr(self.sonichost, self.multi_asic_attr)(*module_args, **complex_args)
2023-10-10T22:36:02.7623823Z File "/azp/_work/71/s/tests/common/devices/base.py", line 78, in _run
2023-10-10T22:36:02.7625521Z res = self.module(*module_args, **complex_args)[self.hostname]
2023-10-10T22:36:02.7627548Z File "/usr/local/lib/python2.7/dist-packages/pytest_ansible/module_dispatcher/v28.py", line 159, in _run
2023-10-10T22:36:02.7628921Z raise AnsibleConnectionFailure("Host unreachable", dark=cb.unreachable, contacted=cb.contacted)
2023-10-10T22:36:02.7630092Z AnsibleConnectionFailure: Host unreachable
2023-10-10T22:36:02.7630937Z 
2023-10-10T22:36:02.8098861Z 
2023-10-10T22:43:41.3783751Z platform_tests/test_reload_config.py::test_reload_configuration[str3-chassis-lc3-1] PASSED [ 83%]
2023-10-10T22:57:53.4785470Z platform_tests/test_reload_config.py::test_reload_configuration_checks[str3-chassis-lc3-1] PASSED [100%]

After the fix:

platform_tests/test_reload_config.py::test_reload_configuration[str3-chassis-sup-2] PASSED [ 16%]
platform_tests/test_reload_config.py::test_reload_configuration_checks[str3-chassis-sup-2] PASSED [ 33%]
platform_tests/test_reload_config.py::test_reload_configuration[str3-chassis-lc4-1] PASSED [ 50%]
platform_tests/test_reload_config.py::test_reload_configuration_checks[str3-chassis-lc4-1] PASSED [ 66%]
platform_tests/test_reload_config.py::test_reload_configuration[str3-chassis-lc2-1] PASSED [ 83%]
platform_tests/test_reload_config.py::test_reload_configuration_checks[str3-chassis-lc2-1] PASSED [100%]

------ generated xml file: /var/xxx/sonic-mgmt-int/tests/logs/tr.xml ------
========================= 6 passed in 4449.18 seconds ==========================

[mssonicbld]

Original PR: #10530