Update test_chassis_reboot.py (sonic-net#15422)

Description of PR
Summary:
Fixes # (issue)
Fix the issue when running dut.command("reboot"), when some T2 platform running the command itself can exceed ansible timeout we defined in ansible.cfg that is 60sec.
In this case, test will error out saying "Host unreachable", but it's actually due to ansible command timeout.

Approach
What is the motivation for this PR?
Avoid testcase failure that is command runtime dependent

How did you do it?
Thread the reboot command, so even it timeout in a single thread, main thread is fine

How did you verify/test it?
Before:

03/11/2024 12:04:25 test_chassis_reboot.chassis_cold_reboot  L0027 INFO   | Run cold reboot on <MultiAsicSonicHost 8800-lc4>
03/11/2024 12:04:25 base._run                                L0071 DEBUG  | /var/src/sonic-mgmt_8800-1_66b4a53de4614bccc2e74f8c/tests/common/devices/multi_asic.py::_run_on_asics#135: [8800-lc4] AnsibleModule::command, args=["reboot"], kwargs={}
03/11/2024 12:05:24 transport._log                           L1873 DEBUG  | EOF in transport thread
03/11/2024 12:05:24 __init__.pytest_runtest_call             L0040 ERROR  | Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/_pytest/python.py", line 1788, in runtest
    self.ihook.pytest_pyfunc_call(pyfuncitem=self)
  File "/usr/local/lib/python3.8/dist-packages/pluggy/_hooks.py", line 513, in __call__
    return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
  File "/usr/local/lib/python3.8/dist-packages/pluggy/_manager.py", line 120, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/usr/local/lib/python3.8/dist-packages/pluggy/_callers.py", line 139, in _multicall
    raise exception.with_traceback(exception.__traceback__)
  File "/usr/local/lib/python3.8/dist-packages/pluggy/_callers.py", line 103, in _multicall
    res = hook_impl.function(*args)
  File "/usr/local/lib/python3.8/dist-packages/_pytest/python.py", line 194, in pytest_pyfunc_call
    result = testfunction(**testargs)
  File "/var/src/sonic-mgmt_t2-8800-1_66b4a53de4614bccc2e74f8c/tests/platform_tests/test_chassis_reboot.py", line 70, in test_parallel_reboot
    chassis_cold_reboot(dut, localhost)
  File "/var/src/sonic-mgmt_t2-8800-1_66b4a53de4614bccc2e74f8c/tests/platform_tests/test_chassis_reboot.py", line 28, in chassis_cold_reboot
    dut.command("reboot")
  File "/var/src/sonic-mgmt-t2-8800-1_66b4a53de4614bccc2e74f8c/tests/common/devices/multi_asic.py", line 135, in _run_on_asics
    return getattr(self.sonichost, self.multi_asic_attr)(*module_args, **complex_args)
  File "/var/src/sonic-mgmt-t2-8800-1_66b4a53de4614bccc2e74f8c/tests/common/devices/base.py", line 105, in _run
    res = self.module(*module_args, **complex_args)[self.hostname]
  File "/usr/local/lib/python3.8/dist-packages/pytest_ansible/module_dispatcher/v213.py", line 232, in _run
    raise AnsibleConnectionFailure(
pytest_ansible.errors.AnsibleConnectionFailure: Host unreachable in the inventory
After:

----------------------------------------------------------------------------------------- live log sessionfinish -----------------------------------------------------------------------------------------05:47:44 __init__.pytest_terminal_summary         L0067 INFO   | Can not get Allure report URL. Please check logs
=============================================================================== 1 passed, 1 warning in 2017.89s (0:33:37) ================================================================================DEBUG:tests.conftest:[log_custom_msg] item: <Function test_parallel_reboot>
DEBUG:tests.conftest:append custom_msg: {'dut_check_result': {'core_dump_check_pass': True, 'config_db_check_pass': False}}
Any platform specific information?
The issue is seen on Cisco T2 that takes more time to reboot. But is a general enhancement.

co-authorized by: [email protected]

Author: wenyiz2021

tests/platform_tests/test_chassis_reboot.py

@@ -5,6 +5,7 @@
 import random
 import logging
 import time
+import concurrent.futures
 from tests.common.helpers.assertions import pytest_assert
 from tests.common.utilities import wait_until
 from tests.common.reboot import wait_for_startup,\
@@ -59,15 +60,16 @@ def test_parallel_reboot(duthosts, localhost, conn_graph_facts, xcvr_skip_list):
 
     core_dumps = {}
     # Perform reboot on multiple LCs within 30sec
-    for dut in duthosts:
-        if dut.is_supervisor_node():
-            continue
+    executor = concurrent.futures.ThreadPoolExecutor(max_workers=8)
+    for dut in duthosts.frontend_nodes:
 
         # collect core dump before reboot
         core_dumps[dut.hostname] = get_core_dump(dut)
 
         # Perform cold reboot on all linecards, with an internal within 30sec to mimic a parallel reboot scenario
-        chassis_cold_reboot(dut, localhost)
+        # Change this to threaded reboot, to avoid ansible command timeout in 60sec, we have seen some T2 platform
+        # reboot exceed 60 sec, and causes test to error out
+        executor.submit(chassis_cold_reboot, dut, localhost)
 
         # Wait for 0 ~ 30sec
         rand_interval = random.randint(0, 30)
@@ -88,9 +90,7 @@ def test_parallel_reboot(duthosts, localhost, conn_graph_facts, xcvr_skip_list):
                       "Not all BGP sessions are established on DUT")
 
     # Check if new core dumps are generated
-    for dut in duthosts:
-        if dut.is_supervisor_node():
-            continue
+    for dut in duthosts.frontend_nodes:
         post_core_dump = get_core_dump(dut)
         new_core_dumps = (set(post_core_dump) - set(core_dumps[dut.hostname]))