[sanity check] add BGP to the sanity check list

[yxieca]

Summary:
Fixes # (issue)

Type of change

• Bug fix
• Testbed and Framework(new/improvement)
• Test case(new/improvement)

Approach

What is the motivation for this PR?

Noticed that nightly test left system in a state that all BGP sessions are gone.

admin@str-dx010-acs-1:~$ show ip bgp sum
% No BGP neighbors found

How did you do it?

• Generate warning log when sanity check recovery action is taken.
• Check BGP neighbor health by default.
• Separate the default check list from supported check list.
• Allow disabling check with command line parameters.
• Increase load_config recovery wait time to 120 seconds.
• Address variable used before initialize issue in bgp_facts.py.

Signed-off-by: Ying Xie [email protected]

How did you verify/test it?

• Without the change, when BGP is not healthy, test can still pass:

================================================================================== test session starts ===================================================================================
platform linux2 -- Python 2.7.12, pytest-4.6.5, py-1.8.1, pluggy-0.13.1
ansible: 2.8.7
rootdir: /var/src/sonic-mgmt/tests, inifile: pytest.ini
plugins: ansible-2.2.2, xdist-1.28.0, forked-1.1.3, repeat-0.8.0
collected 1 item

test_interfaces.py::test_interfaces PASSED

• With the change, when BGP is not healthy, recovery action will be taken:
  ================================================================================== test session starts ===================================================================================
  platform linux2 -- Python 2.7.12, pytest-4.6.5, py-1.8.1, pluggy-0.13.1
  ansible: 2.8.7
  rootdir: /var/src/sonic-mgmt/tests, inifile: pytest.ini
  plugins: ansible-2.2.2, xdist-1.28.0, forked-1.1.3, repeat-0.8.0
  collected 1 item

test_interfaces.py::test_interfaces
------------------------------------------------------------------------------------- live log setup -------------------------------------------------------------------------------------
21:25:46 ERROR recover.py:adaptive_recover:80: Restoring {'check_item': 'bgp', 'failed': True} with proposed action config_reload, final action config_reload
PASSED [100%]

• Manually shutdown a session and it will also trigger recovery:
  ================================================================================== test session starts ===================================================================================
  platform linux2 -- Python 2.7.12, pytest-4.6.5, py-1.8.1, pluggy-0.13.1
  ansible: 2.8.7
  rootdir: /var/src/sonic-mgmt/tests, inifile: pytest.ini
  plugins: ansible-2.2.2, xdist-1.28.0, forked-1.1.3, repeat-0.8.0
  collected 1 item

test_interfaces.py::test_interfaces
------------------------------------------------------------------------------------- live log setup -------------------------------------------------------------------------------------
21:47:44 WARNING checks.py:check_bgp_status:115: BGP neighbors down: [u'10.0.0.33']
21:48:19 WARNING recover.py:adaptive_recover:80: Restoring {'check_item': 'bgp', 'failed': True, 'down_neighbors': [u'10.0.0.33']} with proposed action: config_reload, final action: config_reload
PASSED [100%]

[lgtm-com]

This pull request introduces 1 alert when merging d626dd1bb6ad17d772c0d3515210e9d47f2a1ddf into 7ced602 - view on LGTM.com

new alerts:

• 1 for Unused local variable