[action] [PR:10243] Exit pytest with error code 15 if duthosts fixture fails

[mssonicbld]

Description of PR

Summary:
Fixes # (issue)

Type of change

• Bug fix
• Testbed and Framework(new/improvement)
• Test case(new/improvement)

Back port request

• 201911
• 202012
• 202205

Approach

What is the motivation for this PR?

Sometimes, some cases may cause testbed unhealthy, such as previous case do some operations on DUT, it may cause DUT network unreachable, in this case, currently mechanism throw AnsibleConnectionFailure and still run the next test case, actually, all left cases can't be ran, the whole pytest needs to exit, fail pipeline, it saves time and let user know these is something wrong with this DUT now.
This is traceback when DUT host is unreachable.

__________ ERROR at setup of TestAutoTechSupport.test_max_limit[core] __________

enhance_inventory = None
ansible_adhoc = <function init_host_mgr at 0x7f5826304ad0>
tbinfo = {'auto_recover': 'True', 'comment': 'zitingguo', 'conf-name': 'vms64-t1-s6100-1', 'duts': ['str3-s6100-acs-7'], ...}
request = <SubRequest 'duthosts' for <Function test_sanity>>

 @pytest.fixture(name="duthosts", scope="session")
 def fixture_duthosts(enhance_inventory, ansible_adhoc, tbinfo, request):
 """
 @summary: fixture to get DUT hosts defined in testbed.
 @param ansible_adhoc: Fixture provided by the pytest-ansible package.
 Source of the various device objects. It is
 mandatory argument for the class constructors.
 @param tbinfo: fixture provides information about testbed.
 """
> return DutHosts(ansible_adhoc, tbinfo, get_specified_duts(request))

ansible_adhoc = <function init_host_mgr at 0x7f5826304ad0>
enhance_inventory = None
request = <SubRequest 'duthosts' for <Function test_sanity>>
tbinfo = {'auto_recover': 'True', 'comment': 'zitingguo', 'conf-name': 'vms64-t1-s6100-1', 'duts': ['str3-s6100-acs-7'], ...}

conftest.py:334: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
common/devices/duthosts.py:57: in __init__
 for hostname in tbinfo["duts"] if hostname in duts])
common/devices/multi_asic.py:36: in __init__
 self.sonichost = SonicHost(ansible_adhoc, hostname)
common/devices/sonic.py:78: in __init__
 self._os_version = self._get_os_version()
common/devices/sonic.py:319: in _get_os_version
 output = self.command("sonic-cfggen -y /etc/sonic/sonic_version.yml -v build_version")
common/devices/base.py:78: in _run
 res = self.module(*module_args, **complex_args)[self.hostname]
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <pytest_ansible.module_dispatcher.v28.ModuleDispatcherV28 object at 0x7f582501c250>
module_args = ('sonic-cfggen -y /etc/sonic/sonic_version.yml -v build_version',)
complex_args = {'_raw_params': 'sonic-cfggen -y /etc/sonic/sonic_version.yml -v build_version'}
hosts = [str3-s6100-acs-7], no_hosts = False
args = ['pytest-ansible', 'str3-s6100-acs-7', '--connection=smart', '--become', '--become-method=sudo', '--become-user=root', ...]
verbosity = None, verbosity_syntax = '-vvvvv', argument = 'module-path'
arg_value = ['/azp/_work/31/s/ansible/library']
cb = <pytest_ansible.module_dispatcher.v28.ResultAccumulator object at 0x7f58291acad0>
kwargs = {'inventory': <ansible.inventory.manager.InventoryManager object at 0x7f5824f2f350>, 'loader': <ansible.parsing.datalo...ass': None}, 'stdout_callback': <pytest_ansible.module_dispatcher.v28.ResultAccumulator object at 0x7f58291acad0>, ...}

 def _run(self, *module_args, **complex_args):
 """Execute an ansible adhoc command returning the result in a AdhocResult object."""
 # Assemble module argument string
 if module_args:
 complex_args.update(dict(_raw_params=' '.join(module_args)))
 
 # Assert hosts matching the provided pattern exist
 hosts = self.options['inventory_manager'].list_hosts()
 no_hosts = False
 if len(hosts) == 0:
 no_hosts = True
 warnings.warn("provided hosts list is empty, only localhost is available")
 
 self.options['inventory_manager'].subset(self.options.get('subset'))
 hosts = self.options['inventory_manager'].list_hosts(self.options['host_pattern'])
 if len(hosts) == 0 and not no_hosts:
 raise ansible.errors.AnsibleError("Specified hosts and/or --limit does not match any hosts")
 
 # Pass along cli options
 args = ['pytest-ansible']
 verbosity = None
 for verbosity_syntax in ('-v', '-vv', '-vvv', '-vvvv', '-vvvvv'):
 if verbosity_syntax in sys.argv:
 verbosity = verbosity_syntax
 break
 if verbosity is not None:
 args.append(verbosity_syntax)
 args.extend([self.options['host_pattern']])
 for argument in ('connection', 'user', 'become', 'become_method', 'become_user', 'module_path'):
 arg_value = self.options.get(argument)
 argument = argument.replace('_', '-')
 
 if arg_value in (None, False):
 continue
 
 if arg_value is True:
 args.append('--{0}'.format(argument))
 else:
 args.append('--{0}={1}'.format(argument, arg_value))
 
 # Use Ansible's own adhoc cli to parse the fake command line we created and then save it
 # into Ansible's global context
 adhoc = AdHocCLI(args)
 adhoc.parse()
 
 # And now we'll never speak of this again
 del adhoc
 
 # Initialize callback to capture module JSON responses
 cb = ResultAccumulator()
 
 kwargs = dict(
 inventory=self.options['inventory_manager'],
 variable_manager=self.options['variable_manager'],
 loader=self.options['loader'],
 stdout_callback=cb,
 passwords=dict(conn_pass=None, become_pass=None),
 )
 
 # create a pseudo-play to execute the specified module via a single task
 play_ds = dict(
 name="pytest-ansible",
 hosts=self.options['host_pattern'],
 become=self.options.get('become'),
 become_user=self.options.get('become_user'),
 gather_facts='no',
 tasks=[
 dict(
 action=dict(
 module=self.options['module_name'], args=complex_args
 ),
 ),
 ]
 )
 play = Play().load(play_ds, variable_manager=self.options['variable_manager'], loader=self.options['loader'])
 
 # now create a task queue manager to execute the play
 tqm = None
 try:
 tqm = TaskQueueManager(**kwargs)
 tqm.run(play)
 finally:
 if tqm:
 tqm.cleanup()
 
 
 # Raise exception if host(s) unreachable
 # FIXME - if multiple hosts were involved, should an exception be raised?
 if cb.unreachable:
> raise AnsibleConnectionFailure("Host unreachable", dark=cb.unreachable, contacted=cb.contacted)
E AnsibleConnectionFailure: Host unreachable

How did you do it?

Capture exception in duthosts fixture, when DUT becomes unreachable, this is the first failed fixture. set session.exitstatus to 15 and make run_test.sh aware of this failure and exit pipeline early.

How did you verify/test it?

use run_test.sh to test when dut is unreachable.

Any platform specific information?

Supported testbed topology if it's a new test case?

Documentation

[mssonicbld]

Original PR: #10243

[mssonicbld]

/azp run Azure.sonic-mgmt

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).