Exit pytest with error code 15 if duthosts fixture fails

[ZhaohuiS]

Description of PR

Summary:
Fixes # (issue)

Type of change

• Bug fix
• Testbed and Framework(new/improvement)
• Test case(new/improvement)

Back port request

• 201911
• 202012
• 202205

Approach

What is the motivation for this PR?

Sometimes, some cases may cause testbed unhealthy, such as previous case do some operations on DUT, it may cause DUT network unreachable, in this case, currently mechanism throw AnsibleConnectionFailure and still run the next test case, actually, all left cases can't be ran, the whole pytest needs to exit, fail pipeline, it saves time and let user know these is something wrong with this DUT now.
This is traceback when DUT host is unreachable.

__________ ERROR at setup of TestAutoTechSupport.test_max_limit[core] __________

enhance_inventory = None
ansible_adhoc = <function init_host_mgr at 0x7f5826304ad0>
tbinfo = {'auto_recover': 'True', 'comment': 'zitingguo', 'conf-name': 'vms64-t1-s6100-1', 'duts': ['str3-s6100-acs-7'], ...}
request = <SubRequest 'duthosts' for <Function test_sanity>>

    @pytest.fixture(name="duthosts", scope="session")
    def fixture_duthosts(enhance_inventory, ansible_adhoc, tbinfo, request):
        """
        @summary: fixture to get DUT hosts defined in testbed.
        @param ansible_adhoc: Fixture provided by the pytest-ansible package.
            Source of the various device objects. It is
            mandatory argument for the class constructors.
        @param tbinfo: fixture provides information about testbed.
        """
>       return DutHosts(ansible_adhoc, tbinfo, get_specified_duts(request))

ansible_adhoc = <function init_host_mgr at 0x7f5826304ad0>
enhance_inventory = None
request    = <SubRequest 'duthosts' for <Function test_sanity>>
tbinfo     = {'auto_recover': 'True', 'comment': 'zitingguo', 'conf-name': 'vms64-t1-s6100-1', 'duts': ['str3-s6100-acs-7'], ...}

conftest.py:334: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
common/devices/duthosts.py:57: in __init__
    for hostname in tbinfo["duts"] if hostname in duts])
common/devices/multi_asic.py:36: in __init__
    self.sonichost = SonicHost(ansible_adhoc, hostname)
common/devices/sonic.py:78: in __init__
    self._os_version = self._get_os_version()
common/devices/sonic.py:319: in _get_os_version
    output = self.command("sonic-cfggen -y /etc/sonic/sonic_version.yml -v build_version")
common/devices/base.py:78: in _run
    res = self.module(*module_args, **complex_args)[self.hostname]
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <pytest_ansible.module_dispatcher.v28.ModuleDispatcherV28 object at 0x7f582501c250>
module_args = ('sonic-cfggen -y /etc/sonic/sonic_version.yml -v build_version',)
complex_args = {'_raw_params': 'sonic-cfggen -y /etc/sonic/sonic_version.yml -v build_version'}
hosts = [str3-s6100-acs-7], no_hosts = False
args = ['pytest-ansible', 'str3-s6100-acs-7', '--connection=smart', '--become', '--become-method=sudo', '--become-user=root', ...]
verbosity = None, verbosity_syntax = '-vvvvv', argument = 'module-path'
arg_value = ['/azp/_work/31/s/ansible/library']
cb = <pytest_ansible.module_dispatcher.v28.ResultAccumulator object at 0x7f58291acad0>
kwargs = {'inventory': <ansible.inventory.manager.InventoryManager object at 0x7f5824f2f350>, 'loader': <ansible.parsing.datalo...ass': None}, 'stdout_callback': <pytest_ansible.module_dispatcher.v28.ResultAccumulator object at 0x7f58291acad0>, ...}

    def _run(self, *module_args, **complex_args):
        """Execute an ansible adhoc command returning the result in a AdhocResult object."""
        # Assemble module argument string
        if module_args:
            complex_args.update(dict(_raw_params=' '.join(module_args)))
    
        # Assert hosts matching the provided pattern exist
        hosts = self.options['inventory_manager'].list_hosts()
        no_hosts = False
        if len(hosts) == 0:
            no_hosts = True
            warnings.warn("provided hosts list is empty, only localhost is available")
    
        self.options['inventory_manager'].subset(self.options.get('subset'))
        hosts = self.options['inventory_manager'].list_hosts(self.options['host_pattern'])
        if len(hosts) == 0 and not no_hosts:
            raise ansible.errors.AnsibleError("Specified hosts and/or --limit does not match any hosts")
    
        # Pass along cli options
        args = ['pytest-ansible']
        verbosity = None
        for verbosity_syntax in ('-v', '-vv', '-vvv', '-vvvv', '-vvvvv'):
            if verbosity_syntax in sys.argv:
                verbosity = verbosity_syntax
                break
        if verbosity is not None:
            args.append(verbosity_syntax)
        args.extend([self.options['host_pattern']])
        for argument in ('connection', 'user', 'become', 'become_method', 'become_user', 'module_path'):
            arg_value = self.options.get(argument)
            argument = argument.replace('_', '-')
    
            if arg_value in (None, False):
                continue
    
            if arg_value is True:
                args.append('--{0}'.format(argument))
            else:
                args.append('--{0}={1}'.format(argument, arg_value))
    
        # Use Ansible's own adhoc cli to parse the fake command line we created and then save it
        # into Ansible's global context
        adhoc = AdHocCLI(args)
        adhoc.parse()
    
        # And now we'll never speak of this again
        del adhoc
    
        # Initialize callback to capture module JSON responses
        cb = ResultAccumulator()
    
        kwargs = dict(
            inventory=self.options['inventory_manager'],
            variable_manager=self.options['variable_manager'],
            loader=self.options['loader'],
            stdout_callback=cb,
            passwords=dict(conn_pass=None, become_pass=None),
        )
    
        # create a pseudo-play to execute the specified module via a single task
        play_ds = dict(
            name="pytest-ansible",
            hosts=self.options['host_pattern'],
            become=self.options.get('become'),
            become_user=self.options.get('become_user'),
            gather_facts='no',
            tasks=[
                dict(
                    action=dict(
                        module=self.options['module_name'], args=complex_args
                    ),
                ),
            ]
        )
        play = Play().load(play_ds, variable_manager=self.options['variable_manager'], loader=self.options['loader'])
    
        # now create a task queue manager to execute the play
        tqm = None
        try:
            tqm = TaskQueueManager(**kwargs)
            tqm.run(play)
        finally:
            if tqm:
                tqm.cleanup()
    
    
        # Raise exception if host(s) unreachable
        # FIXME - if multiple hosts were involved, should an exception be raised?
        if cb.unreachable:
>           raise AnsibleConnectionFailure("Host unreachable", dark=cb.unreachable, contacted=cb.contacted)
E           AnsibleConnectionFailure: Host unreachable

How did you do it?

Capture exception in duthosts fixture, when DUT becomes unreachable, this is the first failed fixture. set session.exitstatus to 15 and make run_test.sh aware of this failure and exit pipeline early.

How did you verify/test it?

use run_test.sh to test when dut is unreachable.

Any platform specific information?

Supported testbed topology if it's a new test case?

Documentation

[mssonicbld]

@ZhaohuiS PR conflicts with 202012 branch

[mssonicbld]

Cherry-pick PR to 202205: #10261

[mssonicbld]

Cherry-pick PR to 202305: #10262