test_po_cleanup: replace LogAnalyzer with direct syslog grep to fix marker drop

[Pterosaur]

Description of PR

Fix test_po_cleanup_after_reload spurious failure caused by LogAnalyzer marker being dropped under heavy syslog load.

Type of change

• Bug fix
• Testbed and Framework(new/improvement)
• New Test case
  • Skipped for non-supported platforms
• Test case improvement

Back port request

• 202205
• 202305
• 202311
• 202405
• 202411
• 202505
• 202511

Approach

What is the motivation for this PR?

test_po_cleanup_after_reload fails consistently with:

RuntimeError: cannot find marker end-LogAnalyzer-port_channel_cleanup.xxx in /var/log/syslog

Root cause: The test creates heavy CPU load (16 cores running yes) and then calls config_reload, which restarts all containers. The LogAnalyzer context manager places start/end markers through the syslog UDP socket (/dev/log). On scale setups (e.g. sn5640 with 448 port channels), the config_reload generates such heavy syslog traffic from restarting containers that the host rsyslogd UDP receive buffer overflows for minutes (verified >90 seconds), silently dropping marker messages and causing a RuntimeError.

This supersedes PR #22776 which attempted to fix this by writing markers directly to /var/log/syslog, but that approach causes log timestamp ordering issues.

How did you do it?

Replace the LogAnalyzer context manager usage in test_po_cleanup_after_reload with a direct approach:

1. Record the syslog line count (wc -l /var/log/syslog) before config_reload
2. Run config_reload under CPU stress as before
3. After reload, grep the syslog tail for the expected port-channel cleanup log patterns

This is immune to UDP buffer overflow since it reads the syslog file directly via tail -n +N.

Also improved cleanup: use try/finally to ensure CPU stress processes are always cleaned up.

How did you verify/test it?

Ran on vms70-t1-sn5640-3 (str4-sn5640-3, t1-isolated-d56u1-lag topology, 16 vCPUs, 448 port channels):

1. Without fix: Test fails consistently with RuntimeError: cannot find marker end-LogAnalyzer-... (verified)
2. With fix (direct syslog grep, no LogAnalyzer): Passes reliably (1 passed in 14:32)

Intermediate approaches that did NOT work (for context):

• killall yes + time.sleep(5) inside the with loganalyzer: block: 5s not enough for syslog buffer to drain
• killall yes + 90s polling (send logger messages, check if they appear): syslog UDP buffer stays congested >90 seconds

Any platform specific information?

Tested on Mellanox SN5640. Applies to any platform with enough port channels to generate heavy syslog traffic during config_reload.

Documentation

N/A

[mssonicbld]

/azp run

[azure-pipelines]

Azure Pipelines will not run the associated pipelines, because the pull request was updated after the run command was issued. Review the pull request again and issue a new run command.

[linux-foundation-easycla]

The committers listed above are authorized under a signed CLA.

• ✅ login: Copilot / name: Copilot (f7d3a76)
• ✅ login: Pterosaur / name: Ze Gan (f7d3a76)

[mssonicbld]

/azp run