[sonic-mgmt] Fix test container_checker monit flaky issue. by prajjwal-arista · Pull Request #21218 · sonic-net/sonic-mgmt

prajjwal-arista · 2025-11-06T11:37:47Z

Description of PR

Summary:
These changes fix the sonic-mgmt test "container_checker.py" flaky issue by fixing the bugs found in pytest fixture "update_monit_service":

Fixes # 906

Type of change

Back port request

Approach

What is the motivation for this PR?

As per issue 906 test_container_checker test is flaky.

How did you do it?

It was found out from the failing vs passing logs that the flakiness is due to monit not alerting timely about the containers going down.
After code inspection it was found out that pytest fixture "update_monit_service" in test_container_checker.py was incorrectly updating the etc/monit/conf.d/sonic-host file before the test would begin.
The issue was tested to be resolved by :

fixing bug updating monit conf "/etc/monit/conf.d/sonic-host" for container_checker periodicity. (Currently the sed commands are updating the last line rule, which does not correctly update the container_checker alert periodicity rule.)
Adds wait_until with a pytest_assert after monit restart, to ensure tests are running when monit is properly up.

How did you verify/test it?

Running the test test_container_checker multiple times with the changes to see pass at each time.

Any platform specific information?

N/A

Supported testbed topology if it's a new test case?

N/A

Documentation

N/A

mssonicbld · 2025-11-06T11:37:55Z

/azp run

linux-foundation-easycla · 2025-11-06T11:37:55Z

The committers listed above are authorized under a signed CLA.

✅ login: prajjwal-arista / name: Prajjwal Singh (caa4ba8)

azure-pipelines · 2025-11-06T11:38:10Z

Azure Pipelines successfully started running 1 pipeline(s).

These changes fix the sonic-mgmt test "container_checker.py" flaky issue by fixing the following issues in pytest fixture "update_monit_service": - fixing bug updating monit conf "/etc/monit/conf.d/sonic-host" for container_checker periodicity. (Currently the sed commands are updating the last line rule, which does not correctly update the container_checker alert periodicity rule.) - Adds wait_until after monit restart, to ensure tests are running when monit is properly up. related bug: aristanetworks/sonic-qual.msft#906

mssonicbld · 2025-11-06T13:51:48Z

/azp run

azure-pipelines · 2025-11-06T13:52:05Z

Azure Pipelines successfully started running 1 pipeline(s).

lolyu

LGTM

StormLiangMS · 2025-11-13T03:11:39Z

hi @ZhaohuiS This could be a reference for flaky issue we are handling.

StormLiangMS

LGTM

These changes fix the sonic-mgmt test "container_checker.py" flaky issue by fixing the following issues in pytest fixture "update_monit_service": - fixing bug updating monit conf "/etc/monit/conf.d/sonic-host" for container_checker periodicity. (Currently the sed commands are updating the last line rule, which does not correctly update the container_checker alert periodicity rule.) - Adds wait_until after monit restart, to ensure tests are running when monit is properly up. related bug: https://github.com/aristanetworks/sonic-qual.msft/issues/906

mssonicbld · 2025-11-18T01:31:41Z

Cherry-pick PR to 202505: #21337

These changes fix the sonic-mgmt test "container_checker.py" flaky issue by fixing the following issues in pytest fixture "update_monit_service": - fixing bug updating monit conf "/etc/monit/conf.d/sonic-host" for container_checker periodicity. (Currently the sed commands are updating the last line rule, which does not correctly update the container_checker alert periodicity rule.) - Adds wait_until after monit restart, to ensure tests are running when monit is properly up. related bug: https://github.com/aristanetworks/sonic-qual.msft/issues/906

These changes fix the sonic-mgmt test "container_checker.py" flaky issue by fixing the following issues in pytest fixture "update_monit_service": - fixing bug updating monit conf "/etc/monit/conf.d/sonic-host" for container_checker periodicity. (Currently the sed commands are updating the last line rule, which does not correctly update the container_checker alert periodicity rule.) - Adds wait_until after monit restart, to ensure tests are running when monit is properly up. related bug: https://github.com/aristanetworks/sonic-qual.msft/issues/906 Signed-off-by: vikumarks <[email protected]>

These changes fix the sonic-mgmt test "container_checker.py" flaky issue by fixing the following issues in pytest fixture "update_monit_service": - fixing bug updating monit conf "/etc/monit/conf.d/sonic-host" for container_checker periodicity. (Currently the sed commands are updating the last line rule, which does not correctly update the container_checker alert periodicity rule.) - Adds wait_until after monit restart, to ensure tests are running when monit is properly up. related bug: https://github.com/aristanetworks/sonic-qual.msft/issues/906 Signed-off-by: Alberto Villarreal <[email protected]>

These changes fix the sonic-mgmt test "container_checker.py" flaky issue by fixing the following issues in pytest fixture "update_monit_service": - fixing bug updating monit conf "/etc/monit/conf.d/sonic-host" for container_checker periodicity. (Currently the sed commands are updating the last line rule, which does not correctly update the container_checker alert periodicity rule.) - Adds wait_until after monit restart, to ensure tests are running when monit is properly up. related bug: aristanetworks/sonic-qual.msft#906 Signed-off-by: opcoder0 <[email protected]>

These changes fix the sonic-mgmt test "container_checker.py" flaky issue by fixing the following issues in pytest fixture "update_monit_service": - fixing bug updating monit conf "/etc/monit/conf.d/sonic-host" for container_checker periodicity. (Currently the sed commands are updating the last line rule, which does not correctly update the container_checker alert periodicity rule.) - Adds wait_until after monit restart, to ensure tests are running when monit is properly up. related bug: https://github.com/aristanetworks/sonic-qual.msft/issues/906 Signed-off-by: Dan Caugherty <[email protected]>

These changes fix the sonic-mgmt test "container_checker.py" flaky issue by fixing the following issues in pytest fixture "update_monit_service": - fixing bug updating monit conf "/etc/monit/conf.d/sonic-host" for container_checker periodicity. (Currently the sed commands are updating the last line rule, which does not correctly update the container_checker alert periodicity rule.) - Adds wait_until after monit restart, to ensure tests are running when monit is properly up. related bug: https://github.com/aristanetworks/sonic-qual.msft/issues/906 Signed-off-by: Nishanth Sampath Kumar <[email protected]>

These changes fix the sonic-mgmt test "container_checker.py" flaky issue by fixing the following issues in pytest fixture "update_monit_service": - fixing bug updating monit conf "/etc/monit/conf.d/sonic-host" for container_checker periodicity. (Currently the sed commands are updating the last line rule, which does not correctly update the container_checker alert periodicity rule.) - Adds wait_until after monit restart, to ensure tests are running when monit is properly up. related bug: https://github.com/aristanetworks/sonic-qual.msft/issues/906 Signed-off-by: selldinesh <[email protected]>

These changes fix the sonic-mgmt test "container_checker.py" flaky issue by fixing the following issues in pytest fixture "update_monit_service": - fixing bug updating monit conf "/etc/monit/conf.d/sonic-host" for container_checker periodicity. (Currently the sed commands are updating the last line rule, which does not correctly update the container_checker alert periodicity rule.) - Adds wait_until after monit restart, to ensure tests are running when monit is properly up. related bug: https://github.com/aristanetworks/sonic-qual.msft/issues/906

These changes fix the sonic-mgmt test "container_checker.py" flaky issue by fixing the following issues in pytest fixture "update_monit_service": - fixing bug updating monit conf "/etc/monit/conf.d/sonic-host" for container_checker periodicity. (Currently the sed commands are updating the last line rule, which does not correctly update the container_checker alert periodicity rule.) - Adds wait_until after monit restart, to ensure tests are running when monit is properly up. related bug: https://github.com/aristanetworks/sonic-qual.msft/issues/906 Signed-off-by: Saravanan <[email protected]>

These changes fix the sonic-mgmt test "container_checker.py" flaky issue by fixing the following issues in pytest fixture "update_monit_service": - fixing bug updating monit conf "/etc/monit/conf.d/sonic-host" for container_checker periodicity. (Currently the sed commands are updating the last line rule, which does not correctly update the container_checker alert periodicity rule.) - Adds wait_until after monit restart, to ensure tests are running when monit is properly up. related bug: https://github.com/aristanetworks/sonic-qual.msft/issues/906 Signed-off-by: Guy Shemesh <[email protected]>

These changes fix the sonic-mgmt test "container_checker.py" flaky issue by fixing the following issues in pytest fixture "update_monit_service": - fixing bug updating monit conf "/etc/monit/conf.d/sonic-host" for container_checker periodicity. (Currently the sed commands are updating the last line rule, which does not correctly update the container_checker alert periodicity rule.) - Adds wait_until after monit restart, to ensure tests are running when monit is properly up. related bug: https://github.com/aristanetworks/sonic-qual.msft/issues/906 Signed-off-by: Aharon Malkin <[email protected]>

These changes fix the sonic-mgmt test "container_checker.py" flaky issue by fixing the following issues in pytest fixture "update_monit_service": - fixing bug updating monit conf "/etc/monit/conf.d/sonic-host" for container_checker periodicity. (Currently the sed commands are updating the last line rule, which does not correctly update the container_checker alert periodicity rule.) - Adds wait_until after monit restart, to ensure tests are running when monit is properly up. related bug: https://github.com/aristanetworks/sonic-qual.msft/issues/906 Signed-off-by: Guy Shemesh <[email protected]>

These changes fix the sonic-mgmt test "container_checker.py" flaky issue by fixing the following issues in pytest fixture "update_monit_service": - fixing bug updating monit conf "/etc/monit/conf.d/sonic-host" for container_checker periodicity. (Currently the sed commands are updating the last line rule, which does not correctly update the container_checker alert periodicity rule.) - Adds wait_until after monit restart, to ensure tests are running when monit is properly up. related bug: https://github.com/aristanetworks/sonic-qual.msft/issues/906

These changes fix the sonic-mgmt test "container_checker.py" flaky issue by fixing the following issues in pytest fixture "update_monit_service": - fixing bug updating monit conf "/etc/monit/conf.d/sonic-host" for container_checker periodicity. (Currently the sed commands are updating the last line rule, which does not correctly update the container_checker alert periodicity rule.) - Adds wait_until after monit restart, to ensure tests are running when monit is properly up. related bug: https://github.com/aristanetworks/sonic-qual.msft/issues/906 Signed-off-by: YiFan Wang <[email protected]>

These changes fix the sonic-mgmt test "container_checker.py" flaky issue by fixing the following issues in pytest fixture "update_monit_service": - fixing bug updating monit conf "/etc/monit/conf.d/sonic-host" for container_checker periodicity. (Currently the sed commands are updating the last line rule, which does not correctly update the container_checker alert periodicity rule.) - Adds wait_until after monit restart, to ensure tests are running when monit is properly up. related bug: https://github.com/aristanetworks/sonic-qual.msft/issues/906 Signed-off-by: Lakshmi Yarramaneni <[email protected]>

These changes fix the sonic-mgmt test "container_checker.py" flaky issue by fixing the following issues in pytest fixture "update_monit_service": - fixing bug updating monit conf "/etc/monit/conf.d/sonic-host" for container_checker periodicity. (Currently the sed commands are updating the last line rule, which does not correctly update the container_checker alert periodicity rule.) - Adds wait_until after monit restart, to ensure tests are running when monit is properly up. related bug: https://github.com/aristanetworks/sonic-qual.msft/issues/906

These changes fix the sonic-mgmt test "container_checker.py" flaky issue by fixing the following issues in pytest fixture "update_monit_service": - fixing bug updating monit conf "/etc/monit/conf.d/sonic-host" for container_checker periodicity. (Currently the sed commands are updating the last line rule, which does not correctly update the container_checker alert periodicity rule.) - Adds wait_until after monit restart, to ensure tests are running when monit is properly up. related bug: https://github.com/aristanetworks/sonic-qual.msft/issues/906 Signed-off-by: Yael Tzur <[email protected]>

These changes fix the sonic-mgmt test "container_checker.py" flaky issue by fixing the following issues in pytest fixture "update_monit_service": - fixing bug updating monit conf "/etc/monit/conf.d/sonic-host" for container_checker periodicity. (Currently the sed commands are updating the last line rule, which does not correctly update the container_checker alert periodicity rule.) - Adds wait_until after monit restart, to ensure tests are running when monit is properly up. related bug: https://github.com/aristanetworks/sonic-qual.msft/issues/906

These changes fix the sonic-mgmt test "container_checker.py" flaky issue by fixing the following issues in pytest fixture "update_monit_service": - fixing bug updating monit conf "/etc/monit/conf.d/sonic-host" for container_checker periodicity. (Currently the sed commands are updating the last line rule, which does not correctly update the container_checker alert periodicity rule.) - Adds wait_until after monit restart, to ensure tests are running when monit is properly up. related bug: https://github.com/aristanetworks/sonic-qual.msft/issues/906 Signed-off-by: Raghavendran Ramanathan <[email protected]>

…) (sonic-net#21579) [YangModel][asic_sensors] Add ASIC_SENSORS yang models

These changes fix the sonic-mgmt test "container_checker.py" flaky issue by fixing the following issues in pytest fixture "update_monit_service": - fixing bug updating monit conf "/etc/monit/conf.d/sonic-host" for container_checker periodicity. (Currently the sed commands are updating the last line rule, which does not correctly update the container_checker alert periodicity rule.) - Adds wait_until after monit restart, to ensure tests are running when monit is properly up. related bug: https://github.com/aristanetworks/sonic-qual.msft/issues/906 Signed-off-by: Abhishek <[email protected]>

prajjwal-arista force-pushed the fix-msft906-test_container_checker_monit branch from f9b5262 to caa4ba8 Compare November 6, 2025 13:51

lolyu approved these changes Nov 11, 2025

View reviewed changes

StormLiangMS requested a review from ZhaohuiS November 13, 2025 03:11

StormLiangMS approved these changes Nov 13, 2025

View reviewed changes

StormLiangMS merged commit bc87f3a into sonic-net:master Nov 13, 2025
15 checks passed

StormLiangMS added Request for 202505 branch Approved for 202505 branch labels Nov 13, 2025

mssonicbld added the Created PR to 202505 branch label Nov 18, 2025

mssonicbld mentioned this pull request Nov 18, 2025

[action] [PR:21218] [sonic-mgmt] Fix test container_checker monit flaky issue. #21337

Merged

11 tasks

mssonicbld added Included in 202505 branch and removed Created PR to 202505 branch labels Nov 18, 2025

prajjwal-arista deleted the fix-msft906-test_container_checker_monit branch November 18, 2025 08:46

kazinator-arista pushed a commit to kazinator-arista/sonic-mgmt that referenced this pull request Mar 4, 2026

[YangModel][asic_sensors] Add ASIC_SENSORS yang models (sonic-net#21218…

7ace2d0

…) (sonic-net#21579) [YangModel][asic_sensors] Add ASIC_SENSORS yang models

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[sonic-mgmt] Fix test container_checker monit flaky issue. #21218

[sonic-mgmt] Fix test container_checker monit flaky issue. #21218
StormLiangMS merged 1 commit intosonic-net:masterfrom
prajjwal-arista:fix-msft906-test_container_checker_monit

prajjwal-arista commented Nov 6, 2025 •

edited

Loading

Uh oh!

mssonicbld commented Nov 6, 2025

Uh oh!

linux-foundation-easycla bot commented Nov 6, 2025 •

edited

Loading

Uh oh!

azure-pipelines bot commented Nov 6, 2025

Uh oh!

mssonicbld commented Nov 6, 2025

Uh oh!

azure-pipelines bot commented Nov 6, 2025

Uh oh!

lolyu left a comment

Uh oh!

StormLiangMS commented Nov 13, 2025

Uh oh!

StormLiangMS left a comment

Uh oh!

Uh oh!

mssonicbld commented Nov 18, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

prajjwal-arista commented Nov 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of PR

Type of change

Back port request

Approach

What is the motivation for this PR?

How did you do it?

How did you verify/test it?

Any platform specific information?

Supported testbed topology if it's a new test case?

Documentation

Uh oh!

mssonicbld commented Nov 6, 2025

Uh oh!

linux-foundation-easycla bot commented Nov 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

azure-pipelines bot commented Nov 6, 2025

Uh oh!

mssonicbld commented Nov 6, 2025

Uh oh!

azure-pipelines bot commented Nov 6, 2025

Uh oh!

lolyu left a comment

Choose a reason for hiding this comment

Uh oh!

StormLiangMS commented Nov 13, 2025

Uh oh!

StormLiangMS left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mssonicbld commented Nov 18, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

prajjwal-arista commented Nov 6, 2025 •

edited

Loading

linux-foundation-easycla bot commented Nov 6, 2025 •

edited

Loading