Fix test_warm_reboot issues blocking PR merge

[vaibhavhd]

What I did

Added fixes to test_warm_reboot which is consistently failing on master branch.
This PR should unblock the pending PR merges on sonic-swss due to PR test issues.

Why I did it

There are multiple issues that I think should be fixed in the test_warm_reboot.

Changes added in this PR:

1. Failures to assert restore_count during warm-restart of different services. The cause is that sometimes when service is stopped and immediately started back again, the start command is being missed/ignored/denied by supervisord. This is fixed by adding sleep(1) after service stop and before start.
2. Failures in test_routing_WarmRestart: the errors (below snippet) are supposedly coming due to some change that added weight attribute to return value of routes. Due to this if exact match of dict is done, it would fail as the device has more attributes than what are being expected.

>       assert rt_val == {"ifname": "Ethernet0,Ethernet4,Ethernet8", "nexthop": "111.0.0.2,122.0.0.2,133.0.0.2"}
E       AssertionError: assert {'ifname': 'E...ght': '2,2,2'} == {'ifname': 'E....2,133.0.0.2'}
E         Omitting 2 identical items, use -vv to show
E         Left contains 1 more item:
E         {'weight': '2,2,2'}

3. Whenever failure # 1 is triggered, the device is left in bad state (service stopped, and never restarted), wherein when next testcase gets executed, device does not respond the requests from test, and test keeps waiting indefinitely. Due to # 1 being fixed, hung issue should be seen much less.

Future fixes:

1. Fix how testbed setup and teardown is handled per case within test_warm_reboot.
2. Understand why explicit start is needed for some services, rather then waiting for timed auto restart to kick in.
3. Add a timeout for each case so that even if the test is hung, it does not block pipeline for 8+hours.
4. Collect the artifacts from test_warm_reboot for easier debug.

How I verified it
Tested locally and on pipeline.

Details if related

[stephenxs]

Hi @vaibhavhd
I noticed that it took 6 hours to finish swss vs test, which caused it eventually timeout.
is this the WA to avoid this test issue or we need to extend the timeout?
thank you.

[vaibhavhd]

Hi @vaibhavhd I noticed that it took 6 hours to finish swss vs test, which caused it eventually timeout. is this the WA to avoid this test issue or we need to extend the timeout? thank you.

The previous workaround was added just to confirm if without warmboot the test suite passes or not. It did pass.
Now, I have added fixes for test_warm_reboot, and identified some items to be fixed in future PR.

[prsunny]

@vaibhavhd , thanks for fixing.

[vivekrnv]

PR to 202111 (Eg: #2291, #2313, #2303) are also failing/timing out because of test_warm_reboot. Can you backport this fix to 202111?