[run_test.sh] add maxfail option and improve help information

[yxieca]

Summary:
Fixes # (issue)

Type of change

• Bug fix
• Testbed and Framework(new/improvement)
• Test case(new/improvement)

Approach

What is the motivation for this PR?

PR test behavior changed after switching over to use run_tests.sh. It is no longer stop at first failure and causing longer run time for failed tests.

How did you do it?

• Allow setting maxfail for the test.
• Allow disabling auto recovery
• With individual method, if maxfail is set, any failure in a test
  module will cause test to end.
• Explicit about options with parameter.

Signed-off-by: Ying Xie [email protected]

How did you verify/test it?

Don't specify -q 1 option:

yinxi@acs-trusty8:/var/src/sonic-mgmt/tests$ ./run_tests.sh -d str-dx010-acs-1 -n vms3-t1-dx010-1 -i /var/src/sonic-mgmt/ansible/str,/var/src/sonic-mgmt/ansible/veos -u -m group -p /tmp/logs-01 -c copp/test_copp.py
=== Running tests in groups ===

copp/test_copp.py::TestCOPP::test_policer[ARP]
------------------------------------------------------------------------------------- live log setup -------------------------------------------------------------------------------------
19:53:09 WARNING recover.py:recover:92: Try to recover str-dx010-acs-1 using method adaptive
19:53:09 WARNING recover.py:adaptive_recover:80: Restoring {'check_item': 'processes', 'failed': True, 'services_status': {'lldp': True, 'bgp': True, 'pmon': True, 'database': True, 'teamd': True, 'swss': False, 'syncd': True, 'snmp': True}, 'processes_status': {'lldp': {'status': True, 'exited_critical_process': [], 'running_critical_process': [u'lldp-syncd', u'lldpd', u'lldpmgrd']}, 'pmon': {'status': True, 'exited_critical_process': [], 'running_critical_process': [u'psud', u'xcvrd']}, 'database': {'status': True, 'exited_critical_process': [], 'running_critical_process': [u'redis']}, 'snmp': {'status': True, 'exited_critical_process': [], 'running_critical_process': [u'snmp-subagent', u'snmpd']}, 'bgp': {'status': True, 'exited_critical_process': [], 'running_critical_process': [u'bgpcfgd', u'bgpd', u'fpmsyncd', u'staticd', u'zebra']}, 'teamd': {'status': True, 'exited_critical_process': [], 'running_critical_process': [u'teammgrd', u'teamsyncd']}, 'syncd': {'status': True, 'exited_critical_process': [], 'running_critical_process': [u'syncd']}, 'swss': {'status': False, 'exited_critical_process': [u'orchagent'], 'running_critical_process': [u'buffermgrd', u'intfmgrd', u'nbrmgrd', u'neighsyncd', u'portmgrd', u'portsyncd', u'vlanmgrd', u'vrfmgrd', u'vxlanmgrd']}}} with proposed action: config_reload, final action: config_reload
Loading callback plugin json of type stdout, v2.0 from /usr/local/lib/python2.7/dist-packages/ansible/plugins/callback/json.pyc
Loading callback plugin json of type stdout, v2.0 from /usr/local/lib/python2.7/dist-packages/ansible/plugins/callback/json.pyc
FAILED [ 11%]
copp/test_copp.py::TestCOPP::test_policer[IP2ME] PASSED [ 22%]
copp/test_copp.py::TestCOPP::test_policer[SNMP] PASSED [ 33%]
copp/test_copp.py::TestCOPP::test_policer[SSH] PASSED [ 44%]
copp/test_copp.py::TestCOPP::test_no_policer[BGP] PASSED [ 55%]
copp/test_copp.py::TestCOPP::test_no_policer[DHCP] PASSED [ 66%]
copp/test_copp.py::TestCOPP::test_no_policer[LACP] PASSED [ 77%]
copp/test_copp.py::TestCOPP::test_no_policer[LLDP] PASSED [ 88%]
copp/test_copp.py::TestCOPP::test_no_policer[UDLD] PASSED [100%]

======================================================================================== FAILURES ========================================================================================
=================================================================== 1 failed, 8 passed, 3 warnings in 1154.50 seconds ====================================================================

Specify -q 1 option:
yinxi@acs-trusty8:/var/src/sonic-mgmt/tests$ ./run_tests.sh -d str-dx010-acs-1 -n vms3-t1-dx010-1 -i /var/src/sonic-mgmt/ansible/str,/var/src/sonic-mgmt/ansible/veos -u -m group -p /tmp/logs-01 -c copp/test_copp.py -e -x
=== Running tests in groups ===
================================================================================== test session starts ===================================================================================
platform linux2 -- Python 2.7.12, pytest-4.6.5, py-1.8.1, pluggy-0.13.1
ansible: 2.8.7
rootdir: /var/src/sonic-mgmt/tests, inifile: pytest.ini
plugins: ansible-2.2.2, xdist-1.28.0, forked-1.1.3, repeat-0.8.0
collected 9 items

copp/test_copp.py::TestCOPP::test_policer[ARP] Loading callback plugin json of type stdout, v2.0 from /usr/local/lib/python2.7/dist-packages/ansible/plugins/callback/json.pyc
Loading callback plugin json of type stdout, v2.0 from /usr/local/lib/python2.7/dist-packages/ansible/plugins/callback/json.pyc
FAILED [ 11%]
======================================================================================== FAILURES ========================================================================================
========================================================================= 1 failed, 3 warnings in 453.74 seconds =========================================================================

[daall]

Suggested change

	echo " -e <paramters> : specify extra parameter(s) (default: none)"
	echo " -e <parameters> : specify extra parameter(s) (default: none)"

[yxieca]

Thanks! Fixed.

[daall]

Should we mention log vs. cli log level in the tag so it's more clear which is which at a glance?

[yxieca]

Thanks! Fixed. didn't use full wording so that I don't have to move ':' indentation.

[daall]

@lguohan I can't remember off the top of my head - would a failure in announce_routes or neighbor_health trigger a failure in the old/current PR tests?

[yxieca]

This is a minor detail we can come back to later. I'll merge this PR so that you can work on your next change.

[yxieca]

retest vsimage please

[yxieca]

retest vsimage please