[test_route_perf] Fix test_route_perf.

[bingwang-ms]

Signed-off-by: bingwang [email protected]

Description of PR

Summary:
Fixes #2397
There are several problems in test_route_perf

1. DUT is not cleanup if error is raised before cleanup;
2. Error log reported by route_checker may cause test error, but these errors are possible and explainable ;

Oct 27 04:41:11.849379 str-7260cx3-acs-2 ERR route_check.py:  results: { {#012    "Unaccounted_ROUTE_ENTRY_TABLE_entries": [#012        "3000:146c:1::/64", #012        "3000:146d:1::/64", #012        "3000:146e:1::/64", #012        "3000:146f:1::/64", #012        "3000:1470:1::/64", #012        "3000:1471:1::/64", #012        "3000:1565:1::
Oct 27 04:41:11.850044 str-7260cx3-acs-2 ERR route_check.py:  Failed. Look at reported mismatches above
Oct 27 04:42:08.994134 str-7260cx3-acs-2 ERR monit[627]: 'routeCheck' status failed (255) -- results: { {#012    "Unaccounted_ROUTE_ENTRY_TABLE_entries": [#012        "3000:146c:1::/64", #012        "3000:146d:1::/64", #012        "3000:146e:1::/64", #012        "3000:146f:1::/64", #012        "3000:1470:1::/64", #012        "3000:1471:1::/64", #012        "3000:1482:1::/64", #012        "3000:1483:1::/64",
Oct 27 04:47:11.753617 str-7260cx3-acs-2 INFO monit[627]: 'routeCheck' status succeeded (0) -- no output

3. ip neigh add is not able to add neigh to ARP cache even ip neigh del or ip neigh flush is called.
   This PR address these issues.

Type of change

• Bug fix
• Testbed and Framework(new/improvement)
• Test case(new/improvement)

Approach

What is the motivation for this PR?

This PR is to fix #2397.

How did you do it?

1. Use param to generate tests for ipv4/ipv6
2. Run cleanup even test failed
3. Add config reload to do a further cleanup if test failed
4. Ignore ERROR log reported by route_checker
5. Use 'ip neigh replace' in place of 'ip neigh add'

How did you verify/test it?

Verified on Arista-7260cx3.

py.test --inventory ../ansible/str,../ansible/veos --host-pattern str-7260cx3-acs-2 --module-path ../ansible --testbed vms7-t0-7260-2 --testbed_file ../ansible/testbed.csv --junit-xml=tr.xml --log-cli-level warn --collect_techsupport=False --topology=t0,any,util route/test_route_perf.py
/usr/local/lib/python2.7/dist-packages/cryptography/__init__.py:39: CryptographyDeprecationWarning: Python 2 is no longer supported by the Python core team. Support for it is now deprecated in cryptography, and will be removed in a future release.
  CryptographyDeprecationWarning,
========================================================================================= test session starts =========================================================================================
rootdir: /data/Networking-acs-sonic-mgmt/tests, inifile: pytest.ini
plugins: ansible-2.2.2, forked-1.3.0, xdist-1.28.0, html-1.22.1, repeat-0.8.0, metadata-1.10.0
collected 2 items                                                                                                                                                                                     

route/test_route_perf.py::test_perf_add_remove_routes[4] PASSED                                                                                                                                 [ 50%]
route/test_route_perf.py::test_perf_add_remove_routes[6] PASSED                                                                                                                                 [100%]

------------------------------------------------------------------ generated xml file: /data/Networking-acs-sonic-mgmt/tests/tr.xml -------------------------------------------------------------------
===================================================================================== 2 passed in 194.69 seconds ======================================================================================

The update is also verified on DX010-4, which is running a master branch image. The case will fail because of image issue, bug DUT is cleaned up.

Any platform specific information?

No.

Supported testbed topology if it's a new test case?

No.

Documentation

No.

[shi-su]

Thanks for this great work! I did see a device that fails to finish the routes within this time limit and does not have enough resources for an additional 10000 routes. I thought this might not be an image issue. And we probably want to relax the timeout a bit and reduce the default number of routes.

[bingwang-ms]

I also noticed error reported by swssconfig when applying a large config file.

Oct 26 05:40:54.180200 str-dx010-acs-4 NOTICE swss#swssconfig: :- main: Loading config from JSON file:/arp_test.json...
Oct 26 05:41:00.540328 str-dx010-acs-4 ERR swss#swssconfig: :- main: Exception caught: Unable to connect to redis: Cannot assign requested address

Someone has reported the issue here sonic-net/sonic-buildimage#3849 .