Exit pytest with error code 15 if duthosts fixture fails (sonic-net#10243)

What is the motivation for this PR?
Sometimes, some cases may cause testbed unhealthy, such as previous case do some operations on DUT, it may cause DUT network unreachable, in this case, currently mechanism throw AnsibleConnectionFailure and still run the next test case, actually, all left cases can't be ran, the whole pytest needs to exit, fail pipeline, it saves time and let user know these is something wrong with this DUT now.

How did you do it?
Capture exception in duthosts fixture, when DUT becomes unreachable, this is the first failed fixture. set session.exitstatus to 15 and make run_test.sh aware of this failure and exit pipeline early.
Signed-off-by: Zhaohui Sun <zhaohuisun@microsoft.com>

Author: ZhaohuiS

tests/conftest.py

@@ -48,6 +48,7 @@
 from tests.common.cache import FactsCache
 from tests.common.config_reload import config_reload
 from tests.common.connections.console_host import ConsoleHost
+from tests.common.helpers.assertions import pytest_assert as pt_assert
 
 try:
     from tests.macsec import MacsecPlugin
@@ -63,6 +64,8 @@
 logger = logging.getLogger(__name__)
 cache = FactsCache()
 
+DUTHOSTS_FIXTURE_FAILED_RC = 15
+
 pytest_plugins = ('tests.common.plugins.ptfadapter',
                   'tests.common.plugins.ansible_fixtures',
                   'tests.common.plugins.dut_monitor',
@@ -314,6 +317,12 @@ def get_specified_duts(request):
     return duts
 
 
+def pytest_sessionfinish(session, exitstatus):
+    if session.config.cache.get("duthosts_fixture_failed", None):
+        session.config.cache.set("duthosts_fixture_failed", None)
+        session.exitstatus = DUTHOSTS_FIXTURE_FAILED_RC
+
+
 @pytest.fixture(name="duthosts", scope="session")
 def fixture_duthosts(enhance_inventory, ansible_adhoc, tbinfo, request):
     """
@@ -323,7 +332,14 @@ def fixture_duthosts(enhance_inventory, ansible_adhoc, tbinfo, request):
         mandatory argument for the class constructors.
     @param tbinfo: fixture provides information about testbed.
     """
-    return DutHosts(ansible_adhoc, tbinfo, get_specified_duts(request))
+    try:
+        host = DutHosts(ansible_adhoc, tbinfo, get_specified_duts(request))
+        return host
+    except BaseException as e:
+        logger.error("Failed to initialize duthosts.")
+        request.config.cache.set("duthosts_fixture_failed", True)
+        pt_assert(False, "!!!!!!!!!!!!!!!! duthosts fixture failed !!!!!!!!!!!!!!!!"
+                  "Exception: {}".format(repr(e)))
 
 
 @pytest.fixture(scope="session")

tests/run_tests.sh

@@ -320,6 +320,11 @@ function run_individual_tests()
                 echo "=== Sanity check failed for $test_script. Skip rest of the scripts if there is any. ==="
                 return ${ret_code}
             fi
+            # rc 15 means duthosts fixture failed
+            if [ ${ret_code} -eq 15 ]; then
+                echo "=== duthosts fixture failed for $test_script. Skip rest of the scripts if there is any. ==="
+                return ${ret_code}
+            fi
 
             EXIT_CODE=1
             if [[ ${TEST_MAX_FAIL} != 0 ]]; then