[dualtor_io] Collect syslog to debug by lolyu · Pull Request #17722 · sonic-net/sonic-mgmt

lolyu · 2025-03-27T11:52:15Z

Description of PR

Summary:
Fixes # (issue)

Type of change

Back port request

Approach

What is the motivation for this PR?

Let's collect the syslog for the reload/reboot cases.

Signed-off-by: Longxiang Lyu [email protected]

How did you do it?

Run loganalyzer without analyze.

How did you verify/test it?

Run on dualtor testbed.

Any platform specific information?

Supported testbed topology if it's a new test case?

Documentation

Signed-off-by: Longxiang Lyu <[email protected]>

mssonicbld · 2025-03-27T11:52:23Z

/azp run

azure-pipelines · 2025-03-27T11:52:35Z

Azure Pipelines successfully started running 1 pipeline(s).

bingwang-ms · 2025-03-27T15:48:03Z

Any import required to use fixture setup_loganalyzer?

lolyu · 2025-03-28T00:53:15Z

Any import required to use fixture setup_loganalyzer?

It is in the dualtor_io conftest.py file, no need to import.

Signed-off-by: Longxiang Lyu <[email protected]>

mssonicbld · 2025-03-28T08:27:52Z

/azp run

azure-pipelines · 2025-03-28T08:28:03Z

Azure Pipelines successfully started running 1 pipeline(s).

* [test_link_failure] Collect syslog to debug Signed-off-by: Longxiang Lyu <[email protected]>

mssonicbld · 2025-03-29T14:10:13Z

Cherry-pick PR to 202411: #17741

* [test_link_failure] Collect syslog to debug Signed-off-by: Longxiang Lyu <[email protected]>

Fix the following issue: E Exception: start-LogAnalyzer-test_active_tor_reboot_downstream_standby[active-standby].2025-04-22-10:20:35 was not found in /var/log The issue is introduce by PR: #17722. The root cause is, if the dualtor io reboot failure testcases are running over Arista devices, the syslogs doesn't persist through reboot due to /var/log is a tmpfs directory. So loganalyzer fails to find the start marker in this case. Signed-off-by: Longxiang Lyu [email protected] How did you do it? As the primary goal is to collect syslog after reboot, let's change the start marker as the kernel first boot log, so the dualtor io testcase with reboot will be able to collect logs after kernel boot up. How did you verify/test it? dualtor_io/test_tor_failure.py::test_active_tor_reboot_upstream[active-standby] PASSED [100%] ====================================================================== 1 passed, 1 deselected, 2 warnings in 527.94s (0:08:47) ======================================================================= Signed-off-by: Longxiang Lyu <[email protected]>

Fix the following issue: E Exception: start-LogAnalyzer-test_active_tor_reboot_downstream_standby[active-standby].2025-04-22-10:20:35 was not found in /var/log The issue is introduce by PR: sonic-net#17722. The root cause is, if the dualtor io reboot failure testcases are running over Arista devices, the syslogs doesn't persist through reboot due to /var/log is a tmpfs directory. So loganalyzer fails to find the start marker in this case. Signed-off-by: Longxiang Lyu [email protected] How did you do it? As the primary goal is to collect syslog after reboot, let's change the start marker as the kernel first boot log, so the dualtor io testcase with reboot will be able to collect logs after kernel boot up. How did you verify/test it? dualtor_io/test_tor_failure.py::test_active_tor_reboot_upstream[active-standby] PASSED [100%] ====================================================================== 1 passed, 1 deselected, 2 warnings in 527.94s (0:08:47) ======================================================================= Signed-off-by: Longxiang Lyu <[email protected]>

Fix the following issue: E Exception: start-LogAnalyzer-test_active_tor_reboot_downstream_standby[active-standby].2025-04-22-10:20:35 was not found in /var/log The issue is introduce by PR: #17722. The root cause is, if the dualtor io reboot failure testcases are running over Arista devices, the syslogs doesn't persist through reboot due to /var/log is a tmpfs directory. So loganalyzer fails to find the start marker in this case. Signed-off-by: Longxiang Lyu [email protected] How did you do it? As the primary goal is to collect syslog after reboot, let's change the start marker as the kernel first boot log, so the dualtor io testcase with reboot will be able to collect logs after kernel boot up. How did you verify/test it? dualtor_io/test_tor_failure.py::test_active_tor_reboot_upstream[active-standby] PASSED [100%] ====================================================================== 1 passed, 1 deselected, 2 warnings in 527.94s (0:08:47) ======================================================================= Signed-off-by: Longxiang Lyu <[email protected]>

Code sync sonic-net/sonic-mgmt:202411 => 202503 ``` * 6b59eaa (HEAD -> sync/202503, origin/sync/202503) Merge remote-tracking branch 'pub_upstream/202411' into sync/202503 |\ | * c6a94a0 (pub_upstream/202411) Revert "[dualtor_io] Allow duplications for link down downstream I/O (sonic-net#17909)" (sonic-net#18192) | * de454d5 [testARPCompleted] Cleanup ptf ip after test failure (sonic-net#18170) | * 4a3d1d9 [dualtor] Refine `fdb_mac_learning_test.py` (sonic-net#18092) | * 5964a78 [dualtor_io] Fix the start marker not found issue (sonic-net#18096) | * ce40816 Extend LACP time multiplier for advanced-reboot tests with cEOS peers (sonic-net#17964) | * 0e70ba3 adjust port selection in case testQosSaiXonHysteresis for Cisco-8101 (sonic-net#18130) | * 8bb7203 [202411] Restore disable packet aging fixture 202411 (sonic-net#18103) | * 8f6d1a3 Filter out Not Applicable values in command line (sonic-net#18006) | * 9d5de5c Backport t0-118 test configs to 202411 (sonic-net#17983) | * e758401 mark xfail on generic hash test for isolated topo (sonic-net#18071) | * c65ceab [202411][dualtor] Skip pfcwd warm reboot on dualtor (sonic-net#18072) | * c509006 Improve disabling packet aging to support swap_syncd (sonic-net#17728) (sonic-net#17739) | * 9dc2244 [202411][dualtor-aa] Fix test_arp_dualtor on active-active dualtor (sonic-net#18073) | * cf12a33 fixed tacacs duplicate user issue (sonic-net#18068) | * 330a893 Fix telemetry/test_events.py for dualtor (sonic-net#18025) | * dc6fee8 Remove admin down ports in BUFFER PG check logic (sonic-net#17505) | * 805d538 Update generic hash test to support dualtor active active topology (sonic-net#16217) | * 7c31e46 [dualtor_io] Allow duplications for link down downstream I/O (sonic-net#17909) | * a7f50c6 Fix vlan vs router mac issue with test_qos_dscp_mapping.py (sonic-net#17846) (sonic-net#18003) | * 9ab1e7a Skip test_incremental_qos on Mellanox dualtor (sonic-net#17406) (sonic-net#18048) | * f42afd0 Force eos default creds to be string (sonic-net#18026) | * be542b0 Restore config after vxlan_crm from vxlan_ecmp. (sonic-net#17767) | * f0718b9 [Fix for Issue sonic-net#17413] Modified the Tx Rx port id list selection for all to all scenario (sonic-net#17919) | * 3eb4ed4 [dualtor_io] Collect syslog to debug (sonic-net#17722) | * d5bd995 Disable PFC-WD during PCBB and some wmk test improvements (sonic-net#17889) | * 2f512aa Update outer UDP sport range to exclude port 53 (sonic-net#17570) (sonic-net#17798) | * 980b373 skip test_bgp_slb advanced reboot for isolated topo (sonic-net#17470) | * 408bf9e Default the inner dscp to outer dscp map to be 1-1. (sonic-net#17860) | * 37495a1 Add dualtor fixtures to no_traffic test. (sonic-net#17916) | * a13b599 Only print the matched syslog in loganalzyer teardown check, no traceback info printed (sonic-net#17926) | * 6127f29 Revert "Skip test_vnet_decap on Cisco-8000 with 202411 (sonic-net#17776)" (sonic-net#17941) (sonic-net#17942) | * 60274db Increase timeout to 5 in verify_packet_any_port for background traffic (sonic-net#17904) ```

* [test_link_failure] Collect syslog to debug Signed-off-by: Longxiang Lyu <[email protected]>

Fix the following issue: E Exception: start-LogAnalyzer-test_active_tor_reboot_downstream_standby[active-standby].2025-04-22-10:20:35 was not found in /var/log The issue is introduce by PR: sonic-net#17722. The root cause is, if the dualtor io reboot failure testcases are running over Arista devices, the syslogs doesn't persist through reboot due to /var/log is a tmpfs directory. So loganalyzer fails to find the start marker in this case. Signed-off-by: Longxiang Lyu [email protected] How did you do it? As the primary goal is to collect syslog after reboot, let's change the start marker as the kernel first boot log, so the dualtor io testcase with reboot will be able to collect logs after kernel boot up. How did you verify/test it? dualtor_io/test_tor_failure.py::test_active_tor_reboot_upstream[active-standby] PASSED [100%] ====================================================================== 1 passed, 1 deselected, 2 warnings in 527.94s (0:08:47) ======================================================================= Signed-off-by: Longxiang Lyu <[email protected]>

Fix the following issue: E Exception: start-LogAnalyzer-test_active_tor_reboot_downstream_standby[active-standby].2025-04-22-10:20:35 was not found in /var/log The issue is introduce by PR: sonic-net#17722. The root cause is, if the dualtor io reboot failure testcases are running over Arista devices, the syslogs doesn't persist through reboot due to /var/log is a tmpfs directory. So loganalyzer fails to find the start marker in this case. Signed-off-by: Longxiang Lyu [email protected] How did you do it? As the primary goal is to collect syslog after reboot, let's change the start marker as the kernel first boot log, so the dualtor io testcase with reboot will be able to collect logs after kernel boot up. How did you verify/test it? dualtor_io/test_tor_failure.py::test_active_tor_reboot_upstream[active-standby] PASSED [100%] ====================================================================== 1 passed, 1 deselected, 2 warnings in 527.94s (0:08:47) ======================================================================= Signed-off-by: Longxiang Lyu <[email protected]> Signed-off-by: Aharon Malkin <[email protected]>

* [test_link_failure] Collect syslog to debug Signed-off-by: Longxiang Lyu <[email protected]> Signed-off-by: Guy Shemesh <[email protected]>

Fix the following issue: E Exception: start-LogAnalyzer-test_active_tor_reboot_downstream_standby[active-standby].2025-04-22-10:20:35 was not found in /var/log The issue is introduce by PR: sonic-net#17722. The root cause is, if the dualtor io reboot failure testcases are running over Arista devices, the syslogs doesn't persist through reboot due to /var/log is a tmpfs directory. So loganalyzer fails to find the start marker in this case. Signed-off-by: Longxiang Lyu [email protected] How did you do it? As the primary goal is to collect syslog after reboot, let's change the start marker as the kernel first boot log, so the dualtor io testcase with reboot will be able to collect logs after kernel boot up. How did you verify/test it? dualtor_io/test_tor_failure.py::test_active_tor_reboot_upstream[active-standby] PASSED [100%] ====================================================================== 1 passed, 1 deselected, 2 warnings in 527.94s (0:08:47) ======================================================================= Signed-off-by: Longxiang Lyu <[email protected]> Signed-off-by: Guy Shemesh <[email protected]>

* [test_link_failure] Collect syslog to debug Signed-off-by: Longxiang Lyu <[email protected]> Signed-off-by: Guy Shemesh <[email protected]>

Fix the following issue: E Exception: start-LogAnalyzer-test_active_tor_reboot_downstream_standby[active-standby].2025-04-22-10:20:35 was not found in /var/log The issue is introduce by PR: sonic-net#17722. The root cause is, if the dualtor io reboot failure testcases are running over Arista devices, the syslogs doesn't persist through reboot due to /var/log is a tmpfs directory. So loganalyzer fails to find the start marker in this case. Signed-off-by: Longxiang Lyu [email protected] How did you do it? As the primary goal is to collect syslog after reboot, let's change the start marker as the kernel first boot log, so the dualtor io testcase with reboot will be able to collect logs after kernel boot up. How did you verify/test it? dualtor_io/test_tor_failure.py::test_active_tor_reboot_upstream[active-standby] PASSED [100%] ====================================================================== 1 passed, 1 deselected, 2 warnings in 527.94s (0:08:47) ======================================================================= Signed-off-by: Longxiang Lyu <[email protected]> Signed-off-by: Guy Shemesh <[email protected]>

[test_link_failure] Collect syslog to debug

4c04cc7

Signed-off-by: Longxiang Lyu <[email protected]>

lolyu added the Request for 202411 branch label Mar 27, 2025

lolyu requested review from bingwang-ms and yxieca March 27, 2025 11:52

[dualtor_io][tor_failure] Collect syslog to debug

85a065d

Signed-off-by: Longxiang Lyu <[email protected]>

lolyu changed the title ~~[test_link_failure] Collect syslog to debug~~ [dualtor_io] Collect syslog to debug Mar 28, 2025

bingwang-ms approved these changes Mar 28, 2025

View reviewed changes

bingwang-ms merged commit baa9713 into sonic-net:master Mar 28, 2025
11 checks passed

lolyu added the Approved for 202411 branch label Mar 29, 2025

mssonicbld pushed a commit to mssonicbld/sonic-mgmt that referenced this pull request Mar 29, 2025

[dualtor_io] Collect syslog to debug (sonic-net#17722)

87115f2

* [test_link_failure] Collect syslog to debug Signed-off-by: Longxiang Lyu <[email protected]>

mssonicbld added the Created PR to 202411 branch label Mar 29, 2025

mssonicbld mentioned this pull request Mar 29, 2025

[action] [PR:17722] [dualtor_io] Collect syslog to debug #17741

Merged

11 tasks

amulyan7 pushed a commit to amulyan7/sonic-mgmt that referenced this pull request Mar 31, 2025

[dualtor_io] Collect syslog to debug (sonic-net#17722)

d7d0980

* [test_link_failure] Collect syslog to debug Signed-off-by: Longxiang Lyu <[email protected]>

nnelluri-cisco pushed a commit to nnelluri-cisco/sonic-mgmt that referenced this pull request Mar 31, 2025

[dualtor_io] Collect syslog to debug (sonic-net#17722)

c42dc6b

* [test_link_failure] Collect syslog to debug Signed-off-by: Longxiang Lyu <[email protected]>

OriTrabelsi pushed a commit to OriTrabelsi/sonic-mgmt that referenced this pull request Apr 1, 2025

[dualtor_io] Collect syslog to debug (sonic-net#17722)

63b4c34

* [test_link_failure] Collect syslog to debug Signed-off-by: Longxiang Lyu <[email protected]>

mssonicbld pushed a commit that referenced this pull request Apr 16, 2025

[dualtor_io] Collect syslog to debug (#17722)

3eb4ed4

* [test_link_failure] Collect syslog to debug Signed-off-by: Longxiang Lyu <[email protected]>

mssonicbld added Included in 202411 branch and removed Created PR to 202411 branch labels Apr 16, 2025

lolyu mentioned this pull request Apr 23, 2025

[dualtor_io] Fix the start marker not found issue #18096

Merged

11 tasks

mssonicbld mentioned this pull request Apr 29, 2025

[action] [PR:18096] [dualtor_io] Fix the start marker not found issue #18162

Merged

11 tasks

opcoder0 pushed a commit to opcoder0/sonic-mgmt that referenced this pull request Dec 8, 2025

[dualtor_io] Collect syslog to debug (sonic-net#17722)

9c91eda

* [test_link_failure] Collect syslog to debug Signed-off-by: Longxiang Lyu <[email protected]>

congh-nvidia mentioned this pull request Dec 30, 2025

Bug: Loganalyzer sometimes fails in test dualtor_io/test_tor_failure.py #21799

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[dualtor_io] Collect syslog to debug#17722

[dualtor_io] Collect syslog to debug#17722
bingwang-ms merged 2 commits intosonic-net:masterfrom
lolyu:link_failure_syslog

lolyu commented Mar 27, 2025 •

edited

Loading

Uh oh!

mssonicbld commented Mar 27, 2025

Uh oh!

azure-pipelines bot commented Mar 27, 2025

Uh oh!

bingwang-ms commented Mar 27, 2025

Uh oh!

lolyu commented Mar 28, 2025

Uh oh!

mssonicbld commented Mar 28, 2025

Uh oh!

azure-pipelines bot commented Mar 28, 2025

Uh oh!

Uh oh!

mssonicbld commented Mar 29, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

lolyu commented Mar 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of PR

Type of change

Back port request

Approach

What is the motivation for this PR?

How did you do it?

How did you verify/test it?

Any platform specific information?

Supported testbed topology if it's a new test case?

Documentation

Uh oh!

mssonicbld commented Mar 27, 2025

Uh oh!

azure-pipelines bot commented Mar 27, 2025

Uh oh!

bingwang-ms commented Mar 27, 2025

Uh oh!

lolyu commented Mar 28, 2025

Uh oh!

mssonicbld commented Mar 28, 2025

Uh oh!

azure-pipelines bot commented Mar 28, 2025

Uh oh!

Uh oh!

mssonicbld commented Mar 29, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

lolyu commented Mar 27, 2025 •

edited

Loading