[pytest] Test the feature of container checker. by yozhao101 · Pull Request #2852 · sonic-net/sonic-mgmt

yozhao101 · 2021-01-22T21:32:32Z

Description of PR

Summary:
This PR aims to test the feature of container checker and PR link is sonic-net/sonic-buildimage#6251.

Fixes # (issue)

Type of change

Bug fix
Testbed and Framework(new/improvement)
[x ] Test case(new/improvement)

Approach

What is the motivation for this PR?

This PR aims to test the feature of container checker and PR link of container checker is sonic-net/sonic-buildimage#6251.

The script of container_checker was run periodically by Monit and aims to monitor the running status of each container. Currently the auto-restart feature was enabled. If a critical process exited unexpected, the container will be restarted. If the container was restarted 3 times during 20 minutes, then it will not run anymore unless we cleared the flag using the command sudo systemctl reset-failed <container_name> manually.

How did you do it?

This pytest script will test the script container_checker in the following steps:

Stop the containers explicitly.
Check whether the names of stopped containers appear in the Monit alerting message.
Restart the corresponding stopped containers.
Post-check all the critical processes are running and BGP sessions are established.

How did you verify/test it?

I tested this pytest script on a virtual testbed.

Any platform specific information?

N/A

Supported testbed topology if it's a new test case?

N/A

Documentation

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

lguohan · 2021-01-25T02:43:43Z

can you add this to kvmtest.sh list?

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

bingwang-ms · 2021-01-26T02:38:14Z

tests/container_checker/test_container_checker.py

+    alerting_message = duthost.shell("sudo cat /var/log/syslog | grep -m 1 '.*monit.*container_checker'",
+                                     module_ignore_errors=True)
+
+    pytest_assert(len(alerting_message["stdout_lines"]) > 0,


Suggested change

pytest_assert(len(alerting_message["stdout_lines"]) > 0,

pytest_assert(len(stopped_containers_list) == 0 or len(alerting_message["stdout_lines"]) > 0,

Although it's not likely to happen, I think it's better to add this check.

Great suggestion! Will update.

bingwang-ms · 2021-01-26T02:39:49Z

tests/container_checker/test_container_checker.py

+                  "Failed to get Monit alerting message from container_checker!")
+
+    for container_name in stopped_containers_list:
+        if container_name not in alerting_message["stdout_lines"][0]:


Why alerting_message["stdout_lines"][0] ? Always in the first line?

At here, after the running containers except database were stopped for 6 minutes, the script container_checker would write the alerting message into the syslog. So at the line 223, I used the command sudo cat /var/log/syslog | grep -m 1 '.*monit.*container_checker' to try to get the message generated by container_checker. Since this command will only get a single alerting message from syslog, that's why at there I used alerting_message["stdout_lines"][0]. I do not know whether such method is appropriate or not? Any suggestions?

Updated with a flexible method! Please review again!

I see. Thanks!

bingwang-ms · 2021-01-26T02:44:36Z

tests/container_checker/test_container_checker.py

+    return False
+
+
+def clear_failed_flag_and_restart(duthost, container_name):


Same definition is found in test_container_autorestart.py. Could you help to make it a common utility in tests/common/helpers/dut_utils.py ? Thanks

Great suggestion! Will move this function and its related functions to the file tests/common/helpers/dut_utils.py.

lguohan · 2021-01-26T06:55:40Z

can you fix your test failing?

wangxin · 2021-01-26T10:35:39Z

tests/kvmtest.sh

 platform_tests/test_cpu_memory_usage.py \
-bgp/test_bgpmon.py"
+bgp/test_bgpmon.py" \
+container_checker/test_container_checker.py


The above two lines should be:

bgp/test_bgpmon.py \ container_checker/test_container_checker.py"

Otherwise, the kvmtest.sh script will try to run container_checker/test_container_checker.py and will fail.

Good catch! Fixed.

wangxin · 2021-01-26T10:44:49Z

tests/container_checker/test_container_checker.py

+    Args:
+        duthost: Host DUT.
+        container_name: Name of container.
+        should be running: Boolean value.


should be running -> should_be_running

tests/container_checker/test_container_checker.py

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

1.Move the functions such as 'is_container_running(...)' to the file /tests/common/helpers/dut_utils.py. 2.Fix the logic in the function "check_alerting_message(...)". Signed-off-by: Yong Zhao <yozhao@microsoft.com>

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

yozhao101 · 2021-01-27T00:46:39Z

can you fix your test failing?

Fixed.

yozhao101 · 2021-01-27T00:47:01Z

can you add this to kvmtest.sh list?

Added.

bingwang-ms · 2021-01-27T02:42:48Z

LGTM. Let's watch test results in regression test.

This reverts commit e8bfd88.

…AD (sonic-net#15532) #### Why I did it Update submodule sonic-utilities to the latest HEAD ``` 50296b90 - [202012][dhcp-relay] Fix dhcp6relay counter issue (sonic-net#2866) (sonic-net#2873) (Fri Jun 30 18:08:53 2023 +0800) <Yaqiang Zhu> 160030c4 - [202012][dhcp_relay] Add "Reconfigure", "Information-Request", "Malformed" counter options (sonic-net#2844) (Sun Jun 11 20:35:30 2023 -0700) <kellyyeh> e6289ced - [vlan][dhcp_relay] Clear dhcpv6 relay counter while deleting vlan (sonic-net#2852) (Fri Jun 2 18:20:21 2023 +0000) <Yaqiang Zhu> 885082ec - correctly parsing complete ipv6 vnet info (sonic-net#2827) (Thu May 25 06:01:40 2023 +0000) <Keith Lu> ``` ##### Work item tracking - Microsoft ADO 22635770

…atically (sonic-net#15320) src/sonic-utilities * ec472146 - (HEAD -> 202205, origin/202205) fix show interface neighbor expected empty issue (sonic-net#2465) (3 minutes ago) [jcaiMR] * d1f4413c - [vlan][dhcp_relay] Clear dhcpv6 relay counter while deleting vlan (sonic-net#2852) (6 hours ago) [Yaqiang Zhu] * 051f28ce - [db-migrator] Fix hwsku match for 6100 and add errors when hwsku is None (sonic-net#2821) (7 hours ago) [Vaibhav Hemant Dixit]

…atically (sonic-net#15422) #### Why I did it src/sonic-utilities ``` * 1246bc81 - (HEAD -> 202211, origin/202211) [config reload]Config Reload Enhancement (sonic-net#2693) (sonic-net#2863) (2 days ago) [Sudharsan Dhamal Gopalarathnam] * d69aae4d - [vlan][dhcp_relay] Clear dhcpv6 relay counter while deleting vlan (sonic-net#2852) (2 days ago) [Yaqiang Zhu] * 0f6bf8ac - [config]: Dynamically start and stop ndppd (sonic-net#2814) (2 days ago) [Lawrence Lee] * 48a63ff1 - Fix issue: out of range sflow polling interval is accepted and stored in config_db (sonic-net#2847) (2 days ago) [Junchao-Mellanox] ``` #### How I did it #### How to verify it #### Description for the changelog

yozhao101 added 2 commits January 22, 2021 13:22

[pytest] Test the feature of container checker.

d398b5d

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

[pytest] Use pytest.fail(...) instead of pytest.alert(...).

53046ac

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

yozhao101 requested review from bingwang-ms, jleveque and wangxin January 23, 2021 02:07

yozhao101 requested a review from yxieca January 25, 2021 23:43

[pytest] Add testing the status of containers in KVM testbed.

6bde00c

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

bingwang-ms reviewed Jan 26, 2021

View reviewed changes

wangxin requested changes Jan 26, 2021

View reviewed changes

yozhao101 added 4 commits January 26, 2021 09:07

[pytest] Fix the comment and syntax error.

e4daac6

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

[pytest]

a12529d

1.Move the functions such as 'is_container_running(...)' to the file /tests/common/helpers/dut_utils.py. 2.Fix the logic in the function "check_alerting_message(...)". Signed-off-by: Yong Zhao <yozhao@microsoft.com>

[pytest] Remove the blank lines in the comments of functions.

b574f30

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

[pytest] Organize the imports alphabetically.

a0f0804

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

bingwang-ms approved these changes Jan 27, 2021

View reviewed changes

wangxin approved these changes Jan 27, 2021

View reviewed changes

yozhao101 merged commit e8bfd88 into sonic-net:master Jan 27, 2021

yozhao101 deleted the test_container_status branch January 27, 2021 16:48

lguohan added a commit that referenced this pull request Jan 28, 2021

Revert "[pytest] Test the feature of container checker. (#2852)"

93f655a

This reverts commit e8bfd88.

yozhao101 mentioned this pull request Jan 31, 2021

[pytest] Test the feature of container checker. #2890

Merged

2 tasks

	pytest_assert(len(alerting_message["stdout_lines"]) > 0,
	pytest_assert(len(stopped_containers_list) == 0 or len(alerting_message["stdout_lines"]) > 0,

		return False


		def clear_failed_flag_and_restart(duthost, container_name):

Conversation

yozhao101 commented Jan 22, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of PR

Type of change

Approach

What is the motivation for this PR?

How did you do it?

How did you verify/test it?

Any platform specific information?

Supported testbed topology if it's a new test case?

Documentation

Uh oh!

lguohan commented Jan 25, 2021

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bingwang-ms Jan 26, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yozhao101 Jan 26, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lguohan commented Jan 26, 2021

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

yozhao101 commented Jan 27, 2021

Uh oh!

yozhao101 commented Jan 27, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bingwang-ms commented Jan 27, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

yozhao101 commented Jan 22, 2021 •

edited

Loading

bingwang-ms Jan 26, 2021 •

edited

Loading

yozhao101 Jan 26, 2021 •

edited

Loading

yozhao101 commented Jan 27, 2021 •

edited

Loading