[snappi]: Changed order of reboot and pfcwd_disable for reboot cases in pfc_pause_lossless test#18105
Merged
yejianquan merged 2 commits intosonic-net:masterfrom May 9, 2025
Merged
Conversation
Collaborator
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
Contributor
Author
|
@sdszhang, @auspham, @selldinesh for viz. |
Contributor
sdszhang
reviewed
Apr 28, 2025
| tbinfo, # noqa: F811 | ||
| setup_ports_and_dut, # noqa: F811 | ||
| disable_pfcwd, # noqa: F811 | ||
| reboot_duts): # noqa: F811 |
Contributor
There was a problem hiding this comment.
@amitpawar12, as discussed, the order is not guaranteed here. we can either define a new fixture to explicit define the dependency, or add one function level fixture.
… pfc_pause_lossless test
…less script test modified
amitpawar12
force-pushed
the
snappi-issue-18082
branch
from
fb24b51 to
12f034b
Compare
Collaborator
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
Contributor
|
@vmittal-msft can you help to review this PR? |
vmittal-msft
reviewed
May 5, 2025
| reboot_type = request.param | ||
| _, _, snappi_ports = setup_ports_and_dut | ||
| skip_warm_reboot(snappi_ports[0]['duthost'], reboot_type) | ||
| skip_warm_reboot(snappi_ports[1]['duthost'], reboot_type) |
Contributor
There was a problem hiding this comment.
isn't this repetitive ?
vmittal-msft
reviewed
May 5, 2025
|
|
||
| # Convert the list of duthosts into a list of tuples as required for parallel func. | ||
| args = set((snappi_ports[0]['duthost'], snappi_ports[1]['duthost'])) | ||
| parallel_run(save_config_and_reboot, {}, {}, list(args), timeout=900) |
Contributor
There was a problem hiding this comment.
If src and dest dut is same then will this reboot twice ?
vmittal-msft
approved these changes
May 9, 2025
Collaborator
|
Cherry-pick PR to msft-202405: Azure/sonic-mgmt.msft#261 |
mssonicbld
added
Included in msft-202405 Branch
and removed
Created PR to msft-202405 Branch
labels
opcoder0
pushed a commit
to opcoder0/sonic-mgmt
that referenced
this pull request
Dec 8, 2025
…in pfc_pause_lossless test (sonic-net#18105) Description of PR In tests/snappi_tests/pfc/test_pfc_pause_lossless_with_snappi.py, we have reboot testcases. We disable the PFCWD and Credit-WD in the testcase and reboot the the DUT depending upon type of reboot. However, on cold-reboot, the Credit-WD is getting reset back. This causes the lossless test flows on IXIA to continuously send traffic despite of the pause-storm. The run_traffic in traffic_generation.py is trying to determine if the test stops within a stipulated duration, which it does not and throws an assert that flows have not stopped within the given time duration and attempts. Summary: Fixes # (issue) sonic-net#18082 Type of change Bug fix Testbed and Framework(new/improvement) New Test case Skipped for non-supported platforms Test case improvement Back port request 202012 202205 202305 202311 202405 202411 Approach What is the motivation for this PR? The reboot_dut was called after the PFCWD and Credit-WD disable. The reboot caused the Credit-WD to reset back and hence test failed. How did you do it? Created a single fixture called reboot_duts_and_disable_wd to reboot the DUTs and disable Credit and PFC watch-dogs. This fixture is added in snappi_tests/files/helper.py. Modified the testcase - snappi_tests/pfc/test_pfc_pause_lossless_with_snappi.py to incorporate the above fixture. How did you verify/test it? T0/T1 setup verification: AzDevOps@sonic_mgmt_nokia_202411:/data/sonic-mgmt/tests$ date;python3 -m pytest --inventory ../ansible/ixia-sonic --host-pattern ixr-x3b-14 --testbed ixr-aaa-14-t2 --testbed_file ../ansible/testbed.csv --log-cli-level info --log-file-level info --kube_master unset --showlocals -ra --show-capture stdout --junit-xml=/tmp/p.xml --skip_sanity --log-file=/tmp/p.log --cache-clear --topology tgen snappi_tests/pfc/test_pfc_pause_lossless_with_snappi.py -k reboot Thu May 1 13:45:24 UTC 2025 -------------------------- curtailed output ----------------------- snappi_tests/pfc/test_pfc_pause_lossless_with_snappi.py::test_pfc_pause_single_lossless_prio_reboot[multidut_port_info0-cold-ixr-aaa-14|3] --------------------------------------------------------------------------------------------------- live log setup ---------------------------------------------------------------------------------------------------- 13:46:03 __init__.set_default L0053 INFO | Completeness level not set during test execution. Setting to default level: CompletenessLevel.basic 13:46:03 __init__.check_test_completeness L0151 INFO | Test has no defined levels. Continue without test completeness checks 13:46:03 __init__.loganalyzer L0067 INFO | Log analyzer is disabled 13:46:03 __init__.store_fixture_values L0020 INFO | store memory_utilization test_pfc_pause_single_lossless_prio_reboot[multidut_port_info0-cold-ixr-aaa-14|3] 13:46:03 __init__._fixture_generator_decorator L0081 INFO | -------------------- fixture ignore_route_check_for_cisco_8000 setup starts -------------------- 13:46:03 __init__._fixture_generator_decorator L0085 INFO | -------------------- fixture ignore_route_check_for_cisco_8000 setup ends -------------------- 13:46:03 __init__._fixture_generator_decorator L0081 INFO | -------------------- fixture multidut_port_info setup starts -------------------- 13:46:03 __init__._fixture_generator_decorator L0085 INFO | -------------------- fixture multidut_port_info setup ends -------------------- 13:46:03 __init__._fixture_generator_decorator L0081 INFO | -------------------- fixture number_of_tx_rx_ports setup starts -------------------- 13:46:03 __init__._fixture_generator_decorator L0085 INFO | -------------------- fixture number_of_tx_rx_ports setup ends -------------------- 13:46:03 __init__._fixture_generator_decorator L0081 INFO | -------------------- fixture setup_ports_and_dut setup starts -------------------- 13:46:03 helper.setup_ports_and_dut L0111 INFO | Running test for testbed subtype: single-dut-multi-asic-aaa 13:46:04 __init__._fixture_generator_decorator L0085 INFO | -------------------- fixture setup_ports_and_dut setup ends -------------------- 13:46:04 __init__._fixture_generator_decorator L0081 INFO | -------------------- fixture reboot_duts_and_disable_wd setup starts -------------------- 13:46:05 sonic.get_asic_name L1797 INFO | asic: unknown 13:46:05 helper.skip_warm_reboot L0045 INFO | Reboot type cold is supported on broadcom switches 13:46:05 sonic.get_asic_name L1797 INFO | asic: unknown 13:46:05 helper.skip_warm_reboot L0045 INFO | Reboot type cold is supported on broadcom switches 13:46:08 helper.save_config_and_reboot L0210 INFO | Issuing a cold reboot on the dut ixr-aaa-14 13:46:11 reboot.reboot L0271 INFO | Reboot ixr-aaa-14: wait[120], timeout[300] 13:46:11 reboot.reboot L0273 INFO | DUT ixr-aaa-14 create a file /dev/shm/test_reboot before rebooting 13:46:12 reboot.collect_console_log L0566 INFO | start: collect console log 13:46:12 dut_utils.creds_on_dut L0471 INFO | dut ixr-aaa-14 belongs to groups ['ixia-sonic', 'sonic', 'sonic_aaa', 'fanout'] 13:46:12 dut_utils.creds_on_dut L0496 INFO | skip empty var file /data/sonic-mgmt/tests/common/helpers/../../../ansible/group_vars/all/corefile_uploader.yml 13:46:12 transport._log L1873 INFO | Connected (version 2.0, client OpenSSH_9.2p1) 13:46:16 transport._log L1873 INFO | Auth banner: b'Debian GNU/Linux 12 \n \l ' 13:46:16 transport._log L1873 INFO | Authentication (password) failed. 13:46:16 transport._log L1873 INFO | Connected (version 2.0, client OpenSSH_9.2p1) 13:46:16 transport._log L1873 INFO | Auth banner: b'Debian GNU/Linux 12 \n \l ' 13:46:16 transport._log L1873 INFO | Authentication (password) successful! 13:46:17 reboot.try_create_dut_console L0559 WARNING| Fail to create dut console. Please check console config or if console works ro not. 'ManagementIp' 13:46:17 reboot.collect_console_log L0576 WARNING| dut console is not ready, we cannot get log by console 13:46:17 reboot.wait_for_shutdown L0164 INFO | waiting for ssh to drop on ixr-aaa-14 13:46:17 reboot.execute_reboot_command L0205 INFO | rebooting ixr-aaa-14 with command "reboot" 13:46:35 reboot.wait_for_startup L0185 INFO | waiting for ssh to startup on ixr-aaa-14 13:48:05 reboot.wait_for_startup L0197 INFO | ssh has started up on ixr-aaa-14 13:48:05 reboot.reboot L0300 INFO | waiting for switch ixr-aaa-14 to initialize 13:52:31 processes_utils.wait_critical_processes L0078 INFO | Wait until all critical processes are healthy in 600 sec 13:52:31 processes_utils._all_critical_processes_ L0045 INFO | Check critical processes status 13:52:40 reboot.reboot L0330 INFO | cold reboot finished on ixr-aaa-14 13:52:41 reboot.reboot L0333 INFO | DUT ixr-aaa-14 up since 2025-05-01 13:47:31.410000 13:52:41 helper.save_config_and_reboot L0214 INFO | Wait until the system is stable 13:52:47 sonic_asic.check_bgp_session_state L0714 INFO | bgp neighbors that match the state: ['3.3.3.2', '3333::3:2'] on namespace asic0 13:52:47 sonic_asic.check_bgp_session_state L0715 INFO | bgp neighbors to be checked on the state: [] on namespace asic0 13:52:48 sonic_asic.check_bgp_session_state L0714 INFO | bgp neighbors that match the state: ['3.3.3.1', '3333::3:1'] on namespace asic1 13:52:48 sonic_asic.check_bgp_session_state L0715 INFO | bgp neighbors to be checked on the state: [] on namespace asic1 13:52:48 parallel.on_terminate L0085 INFO | process save_config_and_reboot--<MultiAsicSonicHost ixr-aaa-14> terminated with exit code None 13:52:48 parallel.parallel_run L0221 INFO | Completed running processes for target "save_config_and_reboot" in 0:06:43.000484 seconds 13:52:54 __init__._fixture_generator_decorator L0085 INFO | -------------------- fixture reboot_duts_and_disable_wd setup ends -------------------- 13:52:54 __init__.pytest_runtest_setup L0034 INFO | collect memory before test test_pfc_pause_single_lossless_prio_reboot[multidut_port_info0-cold-ixr-aaa-14|3] 13:52:54 __init__.pytest_runtest_setup L0054 INFO | Before test: collected memory_values {'before_test': {}, 'after_test': {}} -------------------------- curtailed output ----------------------- 13:57:03 traffic_generation.run_traffic L0465 INFO | In-flight traffic statistics for flows: ['Test Flow Prio 3', 'Background Flow Prio 1', 'Background Flow Prio 4', 'Background Flow Prio 5', 'Background Flow Prio 6', 'Background Flow Prio 2', 'Background Flow Prio 0'] 13:57:03 traffic_generation.run_traffic L0466 INFO | In-flight TX frames: [70428, 63697318, 63697318, 63697318, 63697318, 63697318, 63697318] 13:57:03 traffic_generation.run_traffic L0467 INFO | In-flight RX frames: [0, 63697318, 63697318, 63697318, 63697318, 63697318, 63697318] 13:57:21 traffic_generation.run_traffic L0468 INFO | DUT polling complete 13:57:21 traffic_generation.run_traffic L0479 INFO | Checking if all flows have stopped. Attempt #1 13:57:22 traffic_generation.run_traffic L0486 INFO | All test and background traffic flows stopped 13:57:24 traffic_generation.run_traffic L0514 INFO | Dumping per-flow statistics 13:57:25 traffic_generation.run_traffic L0516 INFO | Stopping transmit on all remaining flows 13:57:32 snappi_api.info L1132 INFO | Flows stop 6.258s PASSED -------------------------- curtailed output ----------------------- snappi_tests/pfc/test_pfc_pause_lossless_with_snappi.py::test_pfc_pause_multi_lossless_prio_reboot[multidut_port_info0-cold] --------------------------------------------------------------------------------------------------- live log setup ---------------------------------------------------------------------------------------------------- 14:17:56 __init__.set_default L0053 INFO | Completeness level not set during test execution. Setting to default level: CompletenessLevel.basic 14:17:56 __init__.check_test_completeness L0151 INFO | Test has no defined levels. Continue without test completeness checks 14:17:56 __init__.loganalyzer L0067 INFO | Log analyzer is disabled 14:17:56 __init__.store_fixture_values L0020 INFO | store memory_utilization test_pfc_pause_multi_lossless_prio_reboot[multidut_port_info0-cold] 14:17:56 __init__._fixture_generator_decorator L0081 INFO | -------------------- fixture ignore_route_check_for_cisco_8000 setup starts -------------------- 14:17:56 __init__._fixture_generator_decorator L0085 INFO | -------------------- fixture ignore_route_check_for_cisco_8000 setup ends -------------------- 14:17:56 __init__._fixture_generator_decorator L0081 INFO | -------------------- fixture multidut_port_info setup starts -------------------- 14:17:56 __init__._fixture_generator_decorator L0085 INFO | -------------------- fixture multidut_port_info setup ends -------------------- 14:17:56 __init__._fixture_generator_decorator L0081 INFO | -------------------- fixture number_of_tx_rx_ports setup starts -------------------- 14:17:56 __init__._fixture_generator_decorator L0085 INFO | -------------------- fixture number_of_tx_rx_ports setup ends -------------------- 14:17:56 __init__._fixture_generator_decorator L0081 INFO | -------------------- fixture setup_ports_and_dut setup starts -------------------- 14:17:56 helper.setup_ports_and_dut L0111 INFO | Running test for testbed subtype: single-dut-multi-asic-aaa 14:17:57 __init__._fixture_generator_decorator L0085 INFO | -------------------- fixture setup_ports_and_dut setup ends -------------------- 14:17:57 __init__._fixture_generator_decorator L0081 INFO | -------------------- fixture reboot_duts_and_disable_wd setup starts -------------------- 14:17:58 sonic.get_asic_name L1797 INFO | asic: unknown 14:17:58 helper.skip_warm_reboot L0045 INFO | Reboot type cold is supported on broadcom switches 14:17:58 sonic.get_asic_name L1797 INFO | asic: unknown 14:17:58 helper.skip_warm_reboot L0045 INFO | Reboot type cold is supported on broadcom switches 14:18:01 helper.save_config_and_reboot L0210 INFO | Issuing a cold reboot on the dut ixr-aaa-14 14:18:04 reboot.reboot L0271 INFO | Reboot ixr-aaa-14: wait[120], timeout[300] 14:18:04 reboot.reboot L0273 INFO | DUT ixr-aaa-14 create a file /dev/shm/test_reboot before rebooting 14:18:05 reboot.collect_console_log L0566 INFO | start: collect console log 14:18:05 dut_utils.creds_on_dut L0471 INFO | dut ixr-aaa-14 belongs to groups ['ixia-sonic', 'sonic', 'sonic_aaa', 'fanout'] 14:18:05 dut_utils.creds_on_dut L0496 INFO | skip empty var file /data/sonic-mgmt/tests/common/helpers/../../../ansible/group_vars/all/corefile_uploader.yml 14:18:05 transport._log L1873 INFO | Connected (version 2.0, client OpenSSH_9.2p1) 14:18:07 transport._log L1873 INFO | Auth banner: b'Debian GNU/Linux 12 \n \l ' 14:18:07 transport._log L1873 INFO | Authentication (password) failed. 14:18:07 transport._log L1873 INFO | Connected (version 2.0, client OpenSSH_9.2p1) 14:18:07 transport._log L1873 INFO | Auth banner: b'Debian GNU/Linux 12 \n \l ' 14:18:07 transport._log L1873 INFO | Authentication (password) successful! 14:18:08 reboot.try_create_dut_console L0559 WARNING| Fail to create dut console. Please check console config or if console works ro not. 'ManagementIp' 14:18:08 reboot.collect_console_log L0576 WARNING| dut console is not ready, we cannot get log by console 14:18:10 reboot.wait_for_shutdown L0164 INFO | waiting for ssh to drop on ixr-aaa-14 14:18:10 reboot.execute_reboot_command L0205 INFO | rebooting ixr-aaa-14 with command "reboot" 14:18:28 reboot.wait_for_startup L0185 INFO | waiting for ssh to startup on ixr-aaa-14 14:19:56 reboot.wait_for_startup L0197 INFO | ssh has started up on ixr-aaa-14 14:19:56 reboot.reboot L0300 INFO | waiting for switch ixr-aaa-14 to initialize 14:24:22 processes_utils.wait_critical_processes L0078 INFO | Wait until all critical processes are healthy in 600 sec 14:24:22 processes_utils._all_critical_processes_ L0045 INFO | Check critical processes status 14:24:30 reboot.reboot L0330 INFO | cold reboot finished on ixr-aaa-14 14:24:31 reboot.reboot L0333 INFO | DUT ixr-aaa-14 up since 2025-05-01 14:19:23.320000 14:24:31 helper.save_config_and_reboot L0214 INFO | Wait until the system is stable 14:24:37 sonic_asic.check_bgp_session_state L0714 INFO | bgp neighbors that match the state: ['3.3.3.2', '3333::3:2'] on namespace asic0 14:24:37 sonic_asic.check_bgp_session_state L0715 INFO | bgp neighbors to be checked on the state: [] on namespace asic0 14:24:38 sonic_asic.check_bgp_session_state L0714 INFO | bgp neighbors that match the state: ['3.3.3.1', '3333::3:1'] on namespace asic1 14:24:38 sonic_asic.check_bgp_session_state L0715 INFO | bgp neighbors to be checked on the state: [] on namespace asic1 14:24:38 parallel.on_terminate L0085 INFO | process save_config_and_reboot--<MultiAsicSonicHost ixr-aaa-14> terminated with exit code None 14:24:38 parallel.parallel_run L0221 INFO | Completed running processes for target "save_config_and_reboot" in 0:06:39.645613 seconds 14:24:44 __init__._fixture_generator_decorator L0085 INFO | -------------------- fixture reboot_duts_and_disable_wd setup ends -------------------- 14:24:44 __init__.pytest_runtest_setup L0034 INFO | collect memory before test test_pfc_pause_multi_lossless_prio_reboot[multidut_port_info0-cold] 14:24:44 __init__.pytest_runtest_setup L0054 INFO | Before test: collected memory_values {'before_test': {}, 'after_test': {}} -------------------------- curtailed output ----------------------- 14:28:16 traffic_generation.run_traffic L0465 INFO | In-flight traffic statistics for flows: ['Test Flow Prio 3', 'Test Flow Prio 4', 'Background Flow Prio 1', 'Background Flow Prio 5', 'Background Flow Prio 6', 'Background Flow Prio 2', 'Background Flow Prio 0'] 14:28:16 traffic_generation.run_traffic L0466 INFO | In-flight TX frames: [68215, 68215, 116379310, 116379310, 116379310, 116379310, 116379310] 14:28:16 traffic_generation.run_traffic L0467 INFO | In-flight RX frames: [0, 0, 116379310, 116379310, 116379310, 116379310, 116379310] 14:28:49 traffic_generation.run_traffic L0468 INFO | DUT polling complete 14:28:49 traffic_generation.run_traffic L0479 INFO | Checking if all flows have stopped. Attempt #1 14:28:50 traffic_generation.run_traffic L0486 INFO | All test and background traffic flows stopped 14:28:52 traffic_generation.run_traffic L0514 INFO | Dumping per-flow statistics 14:28:53 traffic_generation.run_traffic L0516 INFO | Stopping transmit on all remaining flows 14:29:00 snappi_api.info L1132 INFO | Flows stop 6.248s PASSED T2 Verification: AzDevOps@7c3002ea4994:/data/tests$ date;python3 -m pytest --inventory ../ansible/ixia-sonic --host-pattern board73,board74 --testbed ixre-chassis17-t2 --testbed_file ../ansible/testbed.csv --show-capture=stdout --log-cli-level info --showlocals -ra --allow_recover --junit-xml=/tmp/pfc/p.xml --skip_sanity --log-file=/tmp/pfc/p.log snappi_tests/pfc/test_pfc_pause_lossless_with_snappi.py -k reboot --pdb Wed Apr 30 20:44:41 UTC 2025 --------------- curtailed output ---------------- snappi_tests/pfc/test_pfc_pause_lossless_with_snappi.py::test_pfc_pause_single_lossless_prio_reboot[multidut_port_info0-cold-board71|3] --------------------------------------------------------------------------------------------------- live log setup ---------------------------------------------------------------------------------------------------- 20:45:40 __init__.set_default L0053 INFO | Completeness level not set during test execution. Setting to default level: CompletenessLevel.basic 20:45:40 __init__.check_test_completeness L0151 INFO | Test has no defined levels. Continue without test completeness checks 20:45:40 __init__.loganalyzer L0067 INFO | Log analyzer is disabled 20:45:40 __init__._fixture_generator_decorator L0081 INFO | -------------------- fixture ignore_route_check_for_cisco_8000 setup starts -------------------- 20:45:40 __init__._fixture_generator_decorator L0085 INFO | -------------------- fixture ignore_route_check_for_cisco_8000 setup ends -------------------- 20:45:40 __init__._fixture_generator_decorator L0081 INFO | -------------------- fixture multidut_port_info setup starts -------------------- 20:45:40 __init__._fixture_generator_decorator L0085 INFO | -------------------- fixture multidut_port_info setup ends -------------------- 20:45:40 __init__._fixture_generator_decorator L0081 INFO | -------------------- fixture number_of_tx_rx_ports setup starts -------------------- 20:45:40 __init__._fixture_generator_decorator L0085 INFO | -------------------- fixture number_of_tx_rx_ports setup ends -------------------- 20:45:40 __init__._fixture_generator_decorator L0081 INFO | -------------------- fixture setup_ports_and_dut setup starts -------------------- 20:45:40 helper.setup_ports_and_dut L0115 INFO | Running test for testbed subtype: multi-dut-long-to-short-link 20:45:43 snappi_fixtures.__intf_config_multidut L0858 INFO | Configuring Dut: board73 with port Ethernet16 with IP 20.1.1.2/31 20:45:43 snappi_fixtures.__intf_config_multidut L0878 INFO | Adding ip:20.1.1.2 on port:Ethernet16 on DUT:board73 20:45:44 snappi_fixtures.__intf_config_multidut L0858 INFO | Configuring Dut: board74 with port Ethernet0 with IP 20.1.1.0/31 20:45:44 snappi_fixtures.__intf_config_multidut L0878 INFO | Adding ip:20.1.1.0 on port:Ethernet0 on DUT:board74 20:45:45 __init__._fixture_generator_decorator L0085 INFO | -------------------- fixture setup_ports_and_dut setup ends -------------------- 20:45:45 __init__._fixture_generator_decorator L0081 INFO | -------------------- fixture reboot_duts_and_disable_wd setup starts -------------------- 20:45:46 sonic.get_asic_name L1778 INFO | asic: unknown 20:45:46 helper.skip_warm_reboot L0049 INFO | Reboot type cold is supported on broadcom switches 20:45:46 sonic.get_asic_name L1778 INFO | asic: unknown 20:45:46 helper.skip_warm_reboot L0049 INFO | Reboot type cold is supported on broadcom switches 20:45:49 helper.save_config_and_reboot L0408 INFO | Issuing a cold reboot on the dut board74 20:45:49 helper.save_config_and_reboot L0408 INFO | Issuing a cold reboot on the dut board73 20:45:52 reboot.reboot L0269 INFO | Reboot board74: wait[900], timeout[600] 20:45:52 reboot.reboot L0271 INFO | DUT board74 create a file /dev/shm/test_reboot before rebooting 20:45:52 reboot.reboot L0269 INFO | Reboot board73: wait[900], timeout[600] 20:45:52 reboot.reboot L0271 INFO | DUT board73 create a file /dev/shm/test_reboot before rebooting 20:45:54 reboot.wait_for_shutdown L0162 INFO | waiting for ssh to drop on board74 20:45:54 reboot.execute_reboot_command L0202 INFO | rebooting board74 with command "reboot" 20:45:54 reboot.wait_for_shutdown L0162 INFO | waiting for ssh to drop on board73 20:45:54 reboot.execute_reboot_command L0202 INFO | rebooting board73 with command "reboot" 20:46:15 reboot.wait_for_startup L0183 INFO | waiting for ssh to startup on board74 20:46:16 reboot.wait_for_startup L0183 INFO | waiting for ssh to startup on board73 20:48:38 reboot.wait_for_startup L0194 INFO | ssh has started up on board73 20:48:38 reboot.reboot L0288 INFO | waiting for switch board73 to initialize 20:48:40 reboot.wait_for_startup L0194 INFO | ssh has started up on board74 20:48:40 reboot.reboot L0288 INFO | waiting for switch board74 to initialize 20:52:24 processes_utils.wait_critical_processes L0079 INFO | Wait until all critical processes are healthy in 600 sec 20:52:24 processes_utils._all_critical_processes_ L0045 INFO | Check critical processes status 20:52:25 processes_utils.wait_critical_processes L0079 INFO | Wait until all critical processes are healthy in 600 sec 20:52:25 processes_utils._all_critical_processes_ L0045 INFO | Check critical processes status 20:52:32 processes_utils.get_critical_processes_s L0035 INFO | The status of checking process in container 'lldp0' is: False 20:52:32 processes_utils.get_critical_processes_s L0037 INFO | The processes not running in container 'lldp0' are: '['lldp-syncd', 'lldpmgrd']' 20:52:32 processes_utils.get_critical_processes_s L0035 INFO | The status of checking process in container 'lldp0' is: False 20:52:32 processes_utils.get_critical_processes_s L0037 INFO | The processes not running in container 'lldp0' are: '['lldp-syncd', 'lldpmgrd']' 20:52:52 processes_utils._all_critical_processes_ L0045 INFO | Check critical processes status 20:52:52 processes_utils._all_critical_processes_ L0045 INFO | Check critical processes status 20:52:59 processes_utils.get_critical_processes_s L0035 INFO | The status of checking process in container 'lldp0' is: False 20:52:59 processes_utils.get_critical_processes_s L0037 INFO | The processes not running in container 'lldp0' are: '['lldp-syncd', 'lldpmgrd']' 20:53:00 processes_utils.get_critical_processes_s L0035 INFO | The status of checking process in container 'lldp0' is: False 20:53:00 processes_utils.get_critical_processes_s L0037 INFO | The processes not running in container 'lldp0' are: '['lldp-syncd', 'lldpmgrd']' 20:53:20 processes_utils._all_critical_processes_ L0045 INFO | Check critical processes status 20:53:20 processes_utils._all_critical_processes_ L0045 INFO | Check critical processes status 20:53:27 reboot.reboot L0318 INFO | cold reboot finished on board73 20:53:28 reboot.reboot L0318 INFO | cold reboot finished on board74 20:53:28 reboot.reboot L0321 INFO | DUT board73 up since 2025-04-30 20:47:22.580000 20:53:28 reboot.reboot L0321 INFO | DUT board74 up since 2025-04-30 20:47:19.270000 20:53:29 helper.save_config_and_reboot L0412 INFO | Wait until the system is stable 20:53:29 helper.save_config_and_reboot L0412 INFO | Wait until the system is stable 20:53:35 sonic_asic.check_bgp_session_state L0692 INFO | bgp neighbors that match the state: ['3.3.3.1', '3.3.3.2', '3.3.3.6', '3.3.3.8', '3.3.3.12', '3.3.3.14', '3.3.3.20', '3333::3:1', '3333::3:2', '3333::3:6', '3333::3:8', '3333::3:12', '3333::3:14', '3333::3:20'] on namespace asic0 20:53:35 sonic_asic.check_bgp_session_state L0693 INFO | bgp neighbors to be checked on the state: [] on namespace asic0 20:53:35 sonic_asic.check_bgp_session_state L0692 INFO | bgp neighbors that match the state: ['3.3.3.1', '3.3.3.2', '3.3.3.6', '3.3.3.8', '3.3.3.14', '3.3.3.18', '3.3.3.20', '3333::3:1', '3333::3:2', '3333::3:6', '3333::3:8', '3333::3:14', '3333::3:18', '3333::3:20'] on namespace asic0 20:53:35 sonic_asic.check_bgp_session_state L0693 INFO | bgp neighbors to be checked on the state: [] on namespace asic0 20:53:36 sonic_asic.check_bgp_session_state L0692 INFO | bgp neighbors that match the state: ['3.3.3.1', '3.3.3.2', '3.3.3.6', '3.3.3.8', '3.3.3.12', '3.3.3.14', '3.3.3.18', '3333::3:1', '3333::3:2', '3333::3:6', '3333::3:8', '3333::3:12', '3333::3:14', '3333::3:18'] on namespace asic1 20:53:36 sonic_asic.check_bgp_session_state L0693 INFO | bgp neighbors to be checked on the state: [] on namespace asic1 20:53:36 parallel.on_terminate L0085 INFO | process save_config_and_reboot--<MultiAsicSonicHost board74> terminated with exit code None 20:53:36 sonic_asic.check_bgp_session_state L0692 INFO | bgp neighbors that match the state: ['3.3.3.1', '3.3.3.2', '3.3.3.6', '3.3.3.8', '3.3.3.12', '3.3.3.18', '3.3.3.20', '3333::3:1', '3333::3:2', '3333::3:6', '3333::3:8', '3333::3:12', '3333::3:18', '3333::3:20'] on namespace asic1 20:53:36 sonic_asic.check_bgp_session_state L0693 INFO | bgp neighbors to be checked on the state: [] on namespace asic1 20:53:36 parallel.on_terminate L0085 INFO | process save_config_and_reboot--<MultiAsicSonicHost board73> terminated with exit code None 20:53:36 parallel.parallel_run L0221 INFO | Completed running processes for target "save_config_and_reboot" in 0:07:49.938416 seconds 20:53:50 __init__._fixture_generator_decorator L0085 INFO | -------------------- fixture reboot_duts_and_disable_wd setup ends -------------------- --------------- curtailed output ---------------- PASSED [ 8%] -------------------------------------------------------------------------------------------------- live log teardown -------------------------------------------------------------------------------------------------- 20:57:53 __init__._fixture_generator_decorator L0093 INFO | -------------------- fixture reboot_duts_and_disable_wd teardown starts -------------------- 20:58:06 config_reload.config_reload L0145 INFO | reloading config_db 21:02:39 processes_utils.wait_critical_processes L0079 INFO | Wait until all critical processes are healthy in 600 sec 21:02:39 processes_utils._all_critical_processes_ L0045 INFO | Check critical processes status 21:02:46 processes_utils.get_critical_processes_s L0035 INFO | The status of checking process in container 'lldp0' is: False 21:02:46 processes_utils.get_critical_processes_s L0037 INFO | The processes not running in container 'lldp0' are: '['lldp-syncd', 'lldpmgrd']' 21:03:06 processes_utils._all_critical_processes_ L0045 INFO | Check critical processes status 21:03:14 config_reload.config_reload L0145 INFO | reloading config_db 21:07:44 processes_utils.wait_critical_processes L0079 INFO | Wait until all critical processes are healthy in 600 sec 21:07:44 processes_utils._all_critical_processes_ L0045 INFO | Check critical processes status 21:07:52 processes_utils.get_critical_processes_s L0035 INFO | The status of checking process in container 'lldp0' is: False 21:07:52 processes_utils.get_critical_processes_s L0037 INFO | The processes not running in container 'lldp0' are: '['lldp-syncd', 'lldpmgrd']' 21:08:12 processes_utils._all_critical_processes_ L0045 INFO | Check critical processes status 21:08:19 __init__._fixture_generator_decorator L0102 INFO | -------------------- fixture reboot_duts_and_disable_wd teardown ends -------------------- 21:08:19 __init__._fixture_generator_decorator L0093 INFO | -------------------- fixture setup_ports_and_dut teardown starts -------------------- 21:08:22 snappi_fixtures.__intf_config_multidut L0858 INFO | Configuring Dut: board73 with port Ethernet16 with IP 20.1.1.2/31 21:08:22 snappi_fixtures.__intf_config_multidut L0880 INFO | Removing ip:20.1.1.2 on port:Ethernet16 on DUT:board73 21:08:23 snappi_fixtures.__intf_config_multidut L0858 INFO | Configuring Dut: board74 with port Ethernet0 with IP 20.1.1.0/31 21:08:23 snappi_fixtures.__intf_config_multidut L0880 INFO | Removing ip:20.1.1.0 on port:Ethernet0 on DUT:board74 21:08:24 __init__._fixture_generator_decorator L0102 INFO | -------------------- fixture setup_ports_and_dut teardown ends -------------------- 21:08:24 __init__._fixture_generator_decorator L0093 INFO | -------------------- fixture number_of_tx_rx_ports teardown starts -------------------- 21:08:24 __init__._fixture_generator_decorator L0102 INFO | -------------------- fixture number_of_tx_rx_ports teardown ends -------------------- 21:08:24 __init__._fixture_generator_decorator L0093 INFO | -------------------- fixture multidut_port_info teardown starts -------------------- 21:08:24 __init__._fixture_generator_decorator L0102 INFO | -------------------- fixture multidut_port_info teardown ends -------------------- 21:08:24 __init__._fixture_generator_decorator L0093 INFO | -------------------- fixture ignore_route_check_for_cisco_8000 teardown starts -------------------- 21:08:24 __init__._fixture_generator_decorator L0102 INFO | -------------------- fixture ignore_route_check_for_cisco_8000 teardown ends -------------------- --------------- curtailed output ---------------- ======================================================================= 8 passed, 16 skipped, 20 deselected, 71 warnings in 10116.17s (2:48:36) ======================================================================= INFO:root:Can not get Allure report URL. Please check logs co-authorized by: jianquanye@microsoft.com Signed-off-by: opcoder0 <110003254+opcoder0@users.noreply.github.com>
AharonMalkin
pushed a commit
to AharonMalkin/sonic-mgmt
that referenced
this pull request
Dec 16, 2025
…in pfc_pause_lossless test (sonic-net#18105) Description of PR In tests/snappi_tests/pfc/test_pfc_pause_lossless_with_snappi.py, we have reboot testcases. We disable the PFCWD and Credit-WD in the testcase and reboot the the DUT depending upon type of reboot. However, on cold-reboot, the Credit-WD is getting reset back. This causes the lossless test flows on IXIA to continuously send traffic despite of the pause-storm. The run_traffic in traffic_generation.py is trying to determine if the test stops within a stipulated duration, which it does not and throws an assert that flows have not stopped within the given time duration and attempts. Summary: Fixes # (issue) sonic-net#18082 Type of change Bug fix Testbed and Framework(new/improvement) New Test case Skipped for non-supported platforms Test case improvement Back port request 202012 202205 202305 202311 202405 202411 Approach What is the motivation for this PR? The reboot_dut was called after the PFCWD and Credit-WD disable. The reboot caused the Credit-WD to reset back and hence test failed. How did you do it? Created a single fixture called reboot_duts_and_disable_wd to reboot the DUTs and disable Credit and PFC watch-dogs. This fixture is added in snappi_tests/files/helper.py. Modified the testcase - snappi_tests/pfc/test_pfc_pause_lossless_with_snappi.py to incorporate the above fixture. How did you verify/test it? T0/T1 setup verification: AzDevOps@sonic_mgmt_nokia_202411:/data/sonic-mgmt/tests$ date;python3 -m pytest --inventory ../ansible/ixia-sonic --host-pattern ixr-x3b-14 --testbed ixr-aaa-14-t2 --testbed_file ../ansible/testbed.csv --log-cli-level info --log-file-level info --kube_master unset --showlocals -ra --show-capture stdout --junit-xml=/tmp/p.xml --skip_sanity --log-file=/tmp/p.log --cache-clear --topology tgen snappi_tests/pfc/test_pfc_pause_lossless_with_snappi.py -k reboot Thu May 1 13:45:24 UTC 2025 -------------------------- curtailed output ----------------------- snappi_tests/pfc/test_pfc_pause_lossless_with_snappi.py::test_pfc_pause_single_lossless_prio_reboot[multidut_port_info0-cold-ixr-aaa-14|3] --------------------------------------------------------------------------------------------------- live log setup ---------------------------------------------------------------------------------------------------- 13:46:03 __init__.set_default L0053 INFO | Completeness level not set during test execution. Setting to default level: CompletenessLevel.basic 13:46:03 __init__.check_test_completeness L0151 INFO | Test has no defined levels. Continue without test completeness checks 13:46:03 __init__.loganalyzer L0067 INFO | Log analyzer is disabled 13:46:03 __init__.store_fixture_values L0020 INFO | store memory_utilization test_pfc_pause_single_lossless_prio_reboot[multidut_port_info0-cold-ixr-aaa-14|3] 13:46:03 __init__._fixture_generator_decorator L0081 INFO | -------------------- fixture ignore_route_check_for_cisco_8000 setup starts -------------------- 13:46:03 __init__._fixture_generator_decorator L0085 INFO | -------------------- fixture ignore_route_check_for_cisco_8000 setup ends -------------------- 13:46:03 __init__._fixture_generator_decorator L0081 INFO | -------------------- fixture multidut_port_info setup starts -------------------- 13:46:03 __init__._fixture_generator_decorator L0085 INFO | -------------------- fixture multidut_port_info setup ends -------------------- 13:46:03 __init__._fixture_generator_decorator L0081 INFO | -------------------- fixture number_of_tx_rx_ports setup starts -------------------- 13:46:03 __init__._fixture_generator_decorator L0085 INFO | -------------------- fixture number_of_tx_rx_ports setup ends -------------------- 13:46:03 __init__._fixture_generator_decorator L0081 INFO | -------------------- fixture setup_ports_and_dut setup starts -------------------- 13:46:03 helper.setup_ports_and_dut L0111 INFO | Running test for testbed subtype: single-dut-multi-asic-aaa 13:46:04 __init__._fixture_generator_decorator L0085 INFO | -------------------- fixture setup_ports_and_dut setup ends -------------------- 13:46:04 __init__._fixture_generator_decorator L0081 INFO | -------------------- fixture reboot_duts_and_disable_wd setup starts -------------------- 13:46:05 sonic.get_asic_name L1797 INFO | asic: unknown 13:46:05 helper.skip_warm_reboot L0045 INFO | Reboot type cold is supported on broadcom switches 13:46:05 sonic.get_asic_name L1797 INFO | asic: unknown 13:46:05 helper.skip_warm_reboot L0045 INFO | Reboot type cold is supported on broadcom switches 13:46:08 helper.save_config_and_reboot L0210 INFO | Issuing a cold reboot on the dut ixr-aaa-14 13:46:11 reboot.reboot L0271 INFO | Reboot ixr-aaa-14: wait[120], timeout[300] 13:46:11 reboot.reboot L0273 INFO | DUT ixr-aaa-14 create a file /dev/shm/test_reboot before rebooting 13:46:12 reboot.collect_console_log L0566 INFO | start: collect console log 13:46:12 dut_utils.creds_on_dut L0471 INFO | dut ixr-aaa-14 belongs to groups ['ixia-sonic', 'sonic', 'sonic_aaa', 'fanout'] 13:46:12 dut_utils.creds_on_dut L0496 INFO | skip empty var file /data/sonic-mgmt/tests/common/helpers/../../../ansible/group_vars/all/corefile_uploader.yml 13:46:12 transport._log L1873 INFO | Connected (version 2.0, client OpenSSH_9.2p1) 13:46:16 transport._log L1873 INFO | Auth banner: b'Debian GNU/Linux 12 \\n \\l\n\n' 13:46:16 transport._log L1873 INFO | Authentication (password) failed. 13:46:16 transport._log L1873 INFO | Connected (version 2.0, client OpenSSH_9.2p1) 13:46:16 transport._log L1873 INFO | Auth banner: b'Debian GNU/Linux 12 \\n \\l\n\n' 13:46:16 transport._log L1873 INFO | Authentication (password) successful! 13:46:17 reboot.try_create_dut_console L0559 WARNING| Fail to create dut console. Please check console config or if console works ro not. 'ManagementIp' 13:46:17 reboot.collect_console_log L0576 WARNING| dut console is not ready, we cannot get log by console 13:46:17 reboot.wait_for_shutdown L0164 INFO | waiting for ssh to drop on ixr-aaa-14 13:46:17 reboot.execute_reboot_command L0205 INFO | rebooting ixr-aaa-14 with command "reboot" 13:46:35 reboot.wait_for_startup L0185 INFO | waiting for ssh to startup on ixr-aaa-14 13:48:05 reboot.wait_for_startup L0197 INFO | ssh has started up on ixr-aaa-14 13:48:05 reboot.reboot L0300 INFO | waiting for switch ixr-aaa-14 to initialize 13:52:31 processes_utils.wait_critical_processes L0078 INFO | Wait until all critical processes are healthy in 600 sec 13:52:31 processes_utils._all_critical_processes_ L0045 INFO | Check critical processes status 13:52:40 reboot.reboot L0330 INFO | cold reboot finished on ixr-aaa-14 13:52:41 reboot.reboot L0333 INFO | DUT ixr-aaa-14 up since 2025-05-01 13:47:31.410000 13:52:41 helper.save_config_and_reboot L0214 INFO | Wait until the system is stable 13:52:47 sonic_asic.check_bgp_session_state L0714 INFO | bgp neighbors that match the state: ['3.3.3.2', '3333::3:2'] on namespace asic0 13:52:47 sonic_asic.check_bgp_session_state L0715 INFO | bgp neighbors to be checked on the state: [] on namespace asic0 13:52:48 sonic_asic.check_bgp_session_state L0714 INFO | bgp neighbors that match the state: ['3.3.3.1', '3333::3:1'] on namespace asic1 13:52:48 sonic_asic.check_bgp_session_state L0715 INFO | bgp neighbors to be checked on the state: [] on namespace asic1 13:52:48 parallel.on_terminate L0085 INFO | process save_config_and_reboot--<MultiAsicSonicHost ixr-aaa-14> terminated with exit code None 13:52:48 parallel.parallel_run L0221 INFO | Completed running processes for target "save_config_and_reboot" in 0:06:43.000484 seconds 13:52:54 __init__._fixture_generator_decorator L0085 INFO | -------------------- fixture reboot_duts_and_disable_wd setup ends -------------------- 13:52:54 __init__.pytest_runtest_setup L0034 INFO | collect memory before test test_pfc_pause_single_lossless_prio_reboot[multidut_port_info0-cold-ixr-aaa-14|3] 13:52:54 __init__.pytest_runtest_setup L0054 INFO | Before test: collected memory_values {'before_test': {}, 'after_test': {}} -------------------------- curtailed output ----------------------- 13:57:03 traffic_generation.run_traffic L0465 INFO | In-flight traffic statistics for flows: ['Test Flow Prio 3', 'Background Flow Prio 1', 'Background Flow Prio 4', 'Background Flow Prio 5', 'Background Flow Prio 6', 'Background Flow Prio 2', 'Background Flow Prio 0'] 13:57:03 traffic_generation.run_traffic L0466 INFO | In-flight TX frames: [70428, 63697318, 63697318, 63697318, 63697318, 63697318, 63697318] 13:57:03 traffic_generation.run_traffic L0467 INFO | In-flight RX frames: [0, 63697318, 63697318, 63697318, 63697318, 63697318, 63697318] 13:57:21 traffic_generation.run_traffic L0468 INFO | DUT polling complete 13:57:21 traffic_generation.run_traffic L0479 INFO | Checking if all flows have stopped. Attempt #1 13:57:22 traffic_generation.run_traffic L0486 INFO | All test and background traffic flows stopped 13:57:24 traffic_generation.run_traffic L0514 INFO | Dumping per-flow statistics 13:57:25 traffic_generation.run_traffic L0516 INFO | Stopping transmit on all remaining flows 13:57:32 snappi_api.info L1132 INFO | Flows stop 6.258s PASSED -------------------------- curtailed output ----------------------- snappi_tests/pfc/test_pfc_pause_lossless_with_snappi.py::test_pfc_pause_multi_lossless_prio_reboot[multidut_port_info0-cold] --------------------------------------------------------------------------------------------------- live log setup ---------------------------------------------------------------------------------------------------- 14:17:56 __init__.set_default L0053 INFO | Completeness level not set during test execution. Setting to default level: CompletenessLevel.basic 14:17:56 __init__.check_test_completeness L0151 INFO | Test has no defined levels. Continue without test completeness checks 14:17:56 __init__.loganalyzer L0067 INFO | Log analyzer is disabled 14:17:56 __init__.store_fixture_values L0020 INFO | store memory_utilization test_pfc_pause_multi_lossless_prio_reboot[multidut_port_info0-cold] 14:17:56 __init__._fixture_generator_decorator L0081 INFO | -------------------- fixture ignore_route_check_for_cisco_8000 setup starts -------------------- 14:17:56 __init__._fixture_generator_decorator L0085 INFO | -------------------- fixture ignore_route_check_for_cisco_8000 setup ends -------------------- 14:17:56 __init__._fixture_generator_decorator L0081 INFO | -------------------- fixture multidut_port_info setup starts -------------------- 14:17:56 __init__._fixture_generator_decorator L0085 INFO | -------------------- fixture multidut_port_info setup ends -------------------- 14:17:56 __init__._fixture_generator_decorator L0081 INFO | -------------------- fixture number_of_tx_rx_ports setup starts -------------------- 14:17:56 __init__._fixture_generator_decorator L0085 INFO | -------------------- fixture number_of_tx_rx_ports setup ends -------------------- 14:17:56 __init__._fixture_generator_decorator L0081 INFO | -------------------- fixture setup_ports_and_dut setup starts -------------------- 14:17:56 helper.setup_ports_and_dut L0111 INFO | Running test for testbed subtype: single-dut-multi-asic-aaa 14:17:57 __init__._fixture_generator_decorator L0085 INFO | -------------------- fixture setup_ports_and_dut setup ends -------------------- 14:17:57 __init__._fixture_generator_decorator L0081 INFO | -------------------- fixture reboot_duts_and_disable_wd setup starts -------------------- 14:17:58 sonic.get_asic_name L1797 INFO | asic: unknown 14:17:58 helper.skip_warm_reboot L0045 INFO | Reboot type cold is supported on broadcom switches 14:17:58 sonic.get_asic_name L1797 INFO | asic: unknown 14:17:58 helper.skip_warm_reboot L0045 INFO | Reboot type cold is supported on broadcom switches 14:18:01 helper.save_config_and_reboot L0210 INFO | Issuing a cold reboot on the dut ixr-aaa-14 14:18:04 reboot.reboot L0271 INFO | Reboot ixr-aaa-14: wait[120], timeout[300] 14:18:04 reboot.reboot L0273 INFO | DUT ixr-aaa-14 create a file /dev/shm/test_reboot before rebooting 14:18:05 reboot.collect_console_log L0566 INFO | start: collect console log 14:18:05 dut_utils.creds_on_dut L0471 INFO | dut ixr-aaa-14 belongs to groups ['ixia-sonic', 'sonic', 'sonic_aaa', 'fanout'] 14:18:05 dut_utils.creds_on_dut L0496 INFO | skip empty var file /data/sonic-mgmt/tests/common/helpers/../../../ansible/group_vars/all/corefile_uploader.yml 14:18:05 transport._log L1873 INFO | Connected (version 2.0, client OpenSSH_9.2p1) 14:18:07 transport._log L1873 INFO | Auth banner: b'Debian GNU/Linux 12 \\n \\l\n\n' 14:18:07 transport._log L1873 INFO | Authentication (password) failed. 14:18:07 transport._log L1873 INFO | Connected (version 2.0, client OpenSSH_9.2p1) 14:18:07 transport._log L1873 INFO | Auth banner: b'Debian GNU/Linux 12 \\n \\l\n\n' 14:18:07 transport._log L1873 INFO | Authentication (password) successful! 14:18:08 reboot.try_create_dut_console L0559 WARNING| Fail to create dut console. Please check console config or if console works ro not. 'ManagementIp' 14:18:08 reboot.collect_console_log L0576 WARNING| dut console is not ready, we cannot get log by console 14:18:10 reboot.wait_for_shutdown L0164 INFO | waiting for ssh to drop on ixr-aaa-14 14:18:10 reboot.execute_reboot_command L0205 INFO | rebooting ixr-aaa-14 with command "reboot" 14:18:28 reboot.wait_for_startup L0185 INFO | waiting for ssh to startup on ixr-aaa-14 14:19:56 reboot.wait_for_startup L0197 INFO | ssh has started up on ixr-aaa-14 14:19:56 reboot.reboot L0300 INFO | waiting for switch ixr-aaa-14 to initialize 14:24:22 processes_utils.wait_critical_processes L0078 INFO | Wait until all critical processes are healthy in 600 sec 14:24:22 processes_utils._all_critical_processes_ L0045 INFO | Check critical processes status 14:24:30 reboot.reboot L0330 INFO | cold reboot finished on ixr-aaa-14 14:24:31 reboot.reboot L0333 INFO | DUT ixr-aaa-14 up since 2025-05-01 14:19:23.320000 14:24:31 helper.save_config_and_reboot L0214 INFO | Wait until the system is stable 14:24:37 sonic_asic.check_bgp_session_state L0714 INFO | bgp neighbors that match the state: ['3.3.3.2', '3333::3:2'] on namespace asic0 14:24:37 sonic_asic.check_bgp_session_state L0715 INFO | bgp neighbors to be checked on the state: [] on namespace asic0 14:24:38 sonic_asic.check_bgp_session_state L0714 INFO | bgp neighbors that match the state: ['3.3.3.1', '3333::3:1'] on namespace asic1 14:24:38 sonic_asic.check_bgp_session_state L0715 INFO | bgp neighbors to be checked on the state: [] on namespace asic1 14:24:38 parallel.on_terminate L0085 INFO | process save_config_and_reboot--<MultiAsicSonicHost ixr-aaa-14> terminated with exit code None 14:24:38 parallel.parallel_run L0221 INFO | Completed running processes for target "save_config_and_reboot" in 0:06:39.645613 seconds 14:24:44 __init__._fixture_generator_decorator L0085 INFO | -------------------- fixture reboot_duts_and_disable_wd setup ends -------------------- 14:24:44 __init__.pytest_runtest_setup L0034 INFO | collect memory before test test_pfc_pause_multi_lossless_prio_reboot[multidut_port_info0-cold] 14:24:44 __init__.pytest_runtest_setup L0054 INFO | Before test: collected memory_values {'before_test': {}, 'after_test': {}} -------------------------- curtailed output ----------------------- 14:28:16 traffic_generation.run_traffic L0465 INFO | In-flight traffic statistics for flows: ['Test Flow Prio 3', 'Test Flow Prio 4', 'Background Flow Prio 1', 'Background Flow Prio 5', 'Background Flow Prio 6', 'Background Flow Prio 2', 'Background Flow Prio 0'] 14:28:16 traffic_generation.run_traffic L0466 INFO | In-flight TX frames: [68215, 68215, 116379310, 116379310, 116379310, 116379310, 116379310] 14:28:16 traffic_generation.run_traffic L0467 INFO | In-flight RX frames: [0, 0, 116379310, 116379310, 116379310, 116379310, 116379310] 14:28:49 traffic_generation.run_traffic L0468 INFO | DUT polling complete 14:28:49 traffic_generation.run_traffic L0479 INFO | Checking if all flows have stopped. Attempt #1 14:28:50 traffic_generation.run_traffic L0486 INFO | All test and background traffic flows stopped 14:28:52 traffic_generation.run_traffic L0514 INFO | Dumping per-flow statistics 14:28:53 traffic_generation.run_traffic L0516 INFO | Stopping transmit on all remaining flows 14:29:00 snappi_api.info L1132 INFO | Flows stop 6.248s PASSED T2 Verification: AzDevOps@7c3002ea4994:/data/tests$ date;python3 -m pytest --inventory ../ansible/ixia-sonic --host-pattern board73,board74 --testbed ixre-chassis17-t2 --testbed_file ../ansible/testbed.csv --show-capture=stdout --log-cli-level info --showlocals -ra --allow_recover --junit-xml=/tmp/pfc/p.xml --skip_sanity --log-file=/tmp/pfc/p.log snappi_tests/pfc/test_pfc_pause_lossless_with_snappi.py -k reboot --pdb Wed Apr 30 20:44:41 UTC 2025 --------------- curtailed output ---------------- snappi_tests/pfc/test_pfc_pause_lossless_with_snappi.py::test_pfc_pause_single_lossless_prio_reboot[multidut_port_info0-cold-board71|3] --------------------------------------------------------------------------------------------------- live log setup ---------------------------------------------------------------------------------------------------- 20:45:40 __init__.set_default L0053 INFO | Completeness level not set during test execution. Setting to default level: CompletenessLevel.basic 20:45:40 __init__.check_test_completeness L0151 INFO | Test has no defined levels. Continue without test completeness checks 20:45:40 __init__.loganalyzer L0067 INFO | Log analyzer is disabled 20:45:40 __init__._fixture_generator_decorator L0081 INFO | -------------------- fixture ignore_route_check_for_cisco_8000 setup starts -------------------- 20:45:40 __init__._fixture_generator_decorator L0085 INFO | -------------------- fixture ignore_route_check_for_cisco_8000 setup ends -------------------- 20:45:40 __init__._fixture_generator_decorator L0081 INFO | -------------------- fixture multidut_port_info setup starts -------------------- 20:45:40 __init__._fixture_generator_decorator L0085 INFO | -------------------- fixture multidut_port_info setup ends -------------------- 20:45:40 __init__._fixture_generator_decorator L0081 INFO | -------------------- fixture number_of_tx_rx_ports setup starts -------------------- 20:45:40 __init__._fixture_generator_decorator L0085 INFO | -------------------- fixture number_of_tx_rx_ports setup ends -------------------- 20:45:40 __init__._fixture_generator_decorator L0081 INFO | -------------------- fixture setup_ports_and_dut setup starts -------------------- 20:45:40 helper.setup_ports_and_dut L0115 INFO | Running test for testbed subtype: multi-dut-long-to-short-link 20:45:43 snappi_fixtures.__intf_config_multidut L0858 INFO | Configuring Dut: board73 with port Ethernet16 with IP 20.1.1.2/31 20:45:43 snappi_fixtures.__intf_config_multidut L0878 INFO | Adding ip:20.1.1.2 on port:Ethernet16 on DUT:board73 20:45:44 snappi_fixtures.__intf_config_multidut L0858 INFO | Configuring Dut: board74 with port Ethernet0 with IP 20.1.1.0/31 20:45:44 snappi_fixtures.__intf_config_multidut L0878 INFO | Adding ip:20.1.1.0 on port:Ethernet0 on DUT:board74 20:45:45 __init__._fixture_generator_decorator L0085 INFO | -------------------- fixture setup_ports_and_dut setup ends -------------------- 20:45:45 __init__._fixture_generator_decorator L0081 INFO | -------------------- fixture reboot_duts_and_disable_wd setup starts -------------------- 20:45:46 sonic.get_asic_name L1778 INFO | asic: unknown 20:45:46 helper.skip_warm_reboot L0049 INFO | Reboot type cold is supported on broadcom switches 20:45:46 sonic.get_asic_name L1778 INFO | asic: unknown 20:45:46 helper.skip_warm_reboot L0049 INFO | Reboot type cold is supported on broadcom switches 20:45:49 helper.save_config_and_reboot L0408 INFO | Issuing a cold reboot on the dut board74 20:45:49 helper.save_config_and_reboot L0408 INFO | Issuing a cold reboot on the dut board73 20:45:52 reboot.reboot L0269 INFO | Reboot board74: wait[900], timeout[600] 20:45:52 reboot.reboot L0271 INFO | DUT board74 create a file /dev/shm/test_reboot before rebooting 20:45:52 reboot.reboot L0269 INFO | Reboot board73: wait[900], timeout[600] 20:45:52 reboot.reboot L0271 INFO | DUT board73 create a file /dev/shm/test_reboot before rebooting 20:45:54 reboot.wait_for_shutdown L0162 INFO | waiting for ssh to drop on board74 20:45:54 reboot.execute_reboot_command L0202 INFO | rebooting board74 with command "reboot" 20:45:54 reboot.wait_for_shutdown L0162 INFO | waiting for ssh to drop on board73 20:45:54 reboot.execute_reboot_command L0202 INFO | rebooting board73 with command "reboot" 20:46:15 reboot.wait_for_startup L0183 INFO | waiting for ssh to startup on board74 20:46:16 reboot.wait_for_startup L0183 INFO | waiting for ssh to startup on board73 20:48:38 reboot.wait_for_startup L0194 INFO | ssh has started up on board73 20:48:38 reboot.reboot L0288 INFO | waiting for switch board73 to initialize 20:48:40 reboot.wait_for_startup L0194 INFO | ssh has started up on board74 20:48:40 reboot.reboot L0288 INFO | waiting for switch board74 to initialize 20:52:24 processes_utils.wait_critical_processes L0079 INFO | Wait until all critical processes are healthy in 600 sec 20:52:24 processes_utils._all_critical_processes_ L0045 INFO | Check critical processes status 20:52:25 processes_utils.wait_critical_processes L0079 INFO | Wait until all critical processes are healthy in 600 sec 20:52:25 processes_utils._all_critical_processes_ L0045 INFO | Check critical processes status 20:52:32 processes_utils.get_critical_processes_s L0035 INFO | The status of checking process in container 'lldp0' is: False 20:52:32 processes_utils.get_critical_processes_s L0037 INFO | The processes not running in container 'lldp0' are: '['lldp-syncd', 'lldpmgrd']' 20:52:32 processes_utils.get_critical_processes_s L0035 INFO | The status of checking process in container 'lldp0' is: False 20:52:32 processes_utils.get_critical_processes_s L0037 INFO | The processes not running in container 'lldp0' are: '['lldp-syncd', 'lldpmgrd']' 20:52:52 processes_utils._all_critical_processes_ L0045 INFO | Check critical processes status 20:52:52 processes_utils._all_critical_processes_ L0045 INFO | Check critical processes status 20:52:59 processes_utils.get_critical_processes_s L0035 INFO | The status of checking process in container 'lldp0' is: False 20:52:59 processes_utils.get_critical_processes_s L0037 INFO | The processes not running in container 'lldp0' are: '['lldp-syncd', 'lldpmgrd']' 20:53:00 processes_utils.get_critical_processes_s L0035 INFO | The status of checking process in container 'lldp0' is: False 20:53:00 processes_utils.get_critical_processes_s L0037 INFO | The processes not running in container 'lldp0' are: '['lldp-syncd', 'lldpmgrd']' 20:53:20 processes_utils._all_critical_processes_ L0045 INFO | Check critical processes status 20:53:20 processes_utils._all_critical_processes_ L0045 INFO | Check critical processes status 20:53:27 reboot.reboot L0318 INFO | cold reboot finished on board73 20:53:28 reboot.reboot L0318 INFO | cold reboot finished on board74 20:53:28 reboot.reboot L0321 INFO | DUT board73 up since 2025-04-30 20:47:22.580000 20:53:28 reboot.reboot L0321 INFO | DUT board74 up since 2025-04-30 20:47:19.270000 20:53:29 helper.save_config_and_reboot L0412 INFO | Wait until the system is stable 20:53:29 helper.save_config_and_reboot L0412 INFO | Wait until the system is stable 20:53:35 sonic_asic.check_bgp_session_state L0692 INFO | bgp neighbors that match the state: ['3.3.3.1', '3.3.3.2', '3.3.3.6', '3.3.3.8', '3.3.3.12', '3.3.3.14', '3.3.3.20', '3333::3:1', '3333::3:2', '3333::3:6', '3333::3:8', '3333::3:12', '3333::3:14', '3333::3:20'] on namespace asic0 20:53:35 sonic_asic.check_bgp_session_state L0693 INFO | bgp neighbors to be checked on the state: [] on namespace asic0 20:53:35 sonic_asic.check_bgp_session_state L0692 INFO | bgp neighbors that match the state: ['3.3.3.1', '3.3.3.2', '3.3.3.6', '3.3.3.8', '3.3.3.14', '3.3.3.18', '3.3.3.20', '3333::3:1', '3333::3:2', '3333::3:6', '3333::3:8', '3333::3:14', '3333::3:18', '3333::3:20'] on namespace asic0 20:53:35 sonic_asic.check_bgp_session_state L0693 INFO | bgp neighbors to be checked on the state: [] on namespace asic0 20:53:36 sonic_asic.check_bgp_session_state L0692 INFO | bgp neighbors that match the state: ['3.3.3.1', '3.3.3.2', '3.3.3.6', '3.3.3.8', '3.3.3.12', '3.3.3.14', '3.3.3.18', '3333::3:1', '3333::3:2', '3333::3:6', '3333::3:8', '3333::3:12', '3333::3:14', '3333::3:18'] on namespace asic1 20:53:36 sonic_asic.check_bgp_session_state L0693 INFO | bgp neighbors to be checked on the state: [] on namespace asic1 20:53:36 parallel.on_terminate L0085 INFO | process save_config_and_reboot--<MultiAsicSonicHost board74> terminated with exit code None 20:53:36 sonic_asic.check_bgp_session_state L0692 INFO | bgp neighbors that match the state: ['3.3.3.1', '3.3.3.2', '3.3.3.6', '3.3.3.8', '3.3.3.12', '3.3.3.18', '3.3.3.20', '3333::3:1', '3333::3:2', '3333::3:6', '3333::3:8', '3333::3:12', '3333::3:18', '3333::3:20'] on namespace asic1 20:53:36 sonic_asic.check_bgp_session_state L0693 INFO | bgp neighbors to be checked on the state: [] on namespace asic1 20:53:36 parallel.on_terminate L0085 INFO | process save_config_and_reboot--<MultiAsicSonicHost board73> terminated with exit code None 20:53:36 parallel.parallel_run L0221 INFO | Completed running processes for target "save_config_and_reboot" in 0:07:49.938416 seconds 20:53:50 __init__._fixture_generator_decorator L0085 INFO | -------------------- fixture reboot_duts_and_disable_wd setup ends -------------------- --------------- curtailed output ---------------- PASSED [ 8%] -------------------------------------------------------------------------------------------------- live log teardown -------------------------------------------------------------------------------------------------- 20:57:53 __init__._fixture_generator_decorator L0093 INFO | -------------------- fixture reboot_duts_and_disable_wd teardown starts -------------------- 20:58:06 config_reload.config_reload L0145 INFO | reloading config_db 21:02:39 processes_utils.wait_critical_processes L0079 INFO | Wait until all critical processes are healthy in 600 sec 21:02:39 processes_utils._all_critical_processes_ L0045 INFO | Check critical processes status 21:02:46 processes_utils.get_critical_processes_s L0035 INFO | The status of checking process in container 'lldp0' is: False 21:02:46 processes_utils.get_critical_processes_s L0037 INFO | The processes not running in container 'lldp0' are: '['lldp-syncd', 'lldpmgrd']' 21:03:06 processes_utils._all_critical_processes_ L0045 INFO | Check critical processes status 21:03:14 config_reload.config_reload L0145 INFO | reloading config_db 21:07:44 processes_utils.wait_critical_processes L0079 INFO | Wait until all critical processes are healthy in 600 sec 21:07:44 processes_utils._all_critical_processes_ L0045 INFO | Check critical processes status 21:07:52 processes_utils.get_critical_processes_s L0035 INFO | The status of checking process in container 'lldp0' is: False 21:07:52 processes_utils.get_critical_processes_s L0037 INFO | The processes not running in container 'lldp0' are: '['lldp-syncd', 'lldpmgrd']' 21:08:12 processes_utils._all_critical_processes_ L0045 INFO | Check critical processes status 21:08:19 __init__._fixture_generator_decorator L0102 INFO | -------------------- fixture reboot_duts_and_disable_wd teardown ends -------------------- 21:08:19 __init__._fixture_generator_decorator L0093 INFO | -------------------- fixture setup_ports_and_dut teardown starts -------------------- 21:08:22 snappi_fixtures.__intf_config_multidut L0858 INFO | Configuring Dut: board73 with port Ethernet16 with IP 20.1.1.2/31 21:08:22 snappi_fixtures.__intf_config_multidut L0880 INFO | Removing ip:20.1.1.2 on port:Ethernet16 on DUT:board73 21:08:23 snappi_fixtures.__intf_config_multidut L0858 INFO | Configuring Dut: board74 with port Ethernet0 with IP 20.1.1.0/31 21:08:23 snappi_fixtures.__intf_config_multidut L0880 INFO | Removing ip:20.1.1.0 on port:Ethernet0 on DUT:board74 21:08:24 __init__._fixture_generator_decorator L0102 INFO | -------------------- fixture setup_ports_and_dut teardown ends -------------------- 21:08:24 __init__._fixture_generator_decorator L0093 INFO | -------------------- fixture number_of_tx_rx_ports teardown starts -------------------- 21:08:24 __init__._fixture_generator_decorator L0102 INFO | -------------------- fixture number_of_tx_rx_ports teardown ends -------------------- 21:08:24 __init__._fixture_generator_decorator L0093 INFO | -------------------- fixture multidut_port_info teardown starts -------------------- 21:08:24 __init__._fixture_generator_decorator L0102 INFO | -------------------- fixture multidut_port_info teardown ends -------------------- 21:08:24 __init__._fixture_generator_decorator L0093 INFO | -------------------- fixture ignore_route_check_for_cisco_8000 teardown starts -------------------- 21:08:24 __init__._fixture_generator_decorator L0102 INFO | -------------------- fixture ignore_route_check_for_cisco_8000 teardown ends -------------------- --------------- curtailed output ---------------- ======================================================================= 8 passed, 16 skipped, 20 deselected, 71 warnings in 10116.17s (2:48:36) ======================================================================= INFO:root:Can not get Allure report URL. Please check logs co-authorized by: jianquanye@microsoft.com Signed-off-by: Aharon Malkin <amalkin@nvidia.com>
gshemesh2
pushed a commit
to gshemesh2/sonic-mgmt
that referenced
this pull request
Dec 21, 2025
…in pfc_pause_lossless test (sonic-net#18105) Description of PR In tests/snappi_tests/pfc/test_pfc_pause_lossless_with_snappi.py, we have reboot testcases. We disable the PFCWD and Credit-WD in the testcase and reboot the the DUT depending upon type of reboot. However, on cold-reboot, the Credit-WD is getting reset back. This causes the lossless test flows on IXIA to continuously send traffic despite of the pause-storm. The run_traffic in traffic_generation.py is trying to determine if the test stops within a stipulated duration, which it does not and throws an assert that flows have not stopped within the given time duration and attempts. Summary: Fixes # (issue) sonic-net#18082 Type of change Bug fix Testbed and Framework(new/improvement) New Test case Skipped for non-supported platforms Test case improvement Back port request 202012 202205 202305 202311 202405 202411 Approach What is the motivation for this PR? The reboot_dut was called after the PFCWD and Credit-WD disable. The reboot caused the Credit-WD to reset back and hence test failed. How did you do it? Created a single fixture called reboot_duts_and_disable_wd to reboot the DUTs and disable Credit and PFC watch-dogs. This fixture is added in snappi_tests/files/helper.py. Modified the testcase - snappi_tests/pfc/test_pfc_pause_lossless_with_snappi.py to incorporate the above fixture. How did you verify/test it? T0/T1 setup verification: AzDevOps@sonic_mgmt_nokia_202411:/data/sonic-mgmt/tests$ date;python3 -m pytest --inventory ../ansible/ixia-sonic --host-pattern ixr-x3b-14 --testbed ixr-aaa-14-t2 --testbed_file ../ansible/testbed.csv --log-cli-level info --log-file-level info --kube_master unset --showlocals -ra --show-capture stdout --junit-xml=/tmp/p.xml --skip_sanity --log-file=/tmp/p.log --cache-clear --topology tgen snappi_tests/pfc/test_pfc_pause_lossless_with_snappi.py -k reboot Thu May 1 13:45:24 UTC 2025 -------------------------- curtailed output ----------------------- snappi_tests/pfc/test_pfc_pause_lossless_with_snappi.py::test_pfc_pause_single_lossless_prio_reboot[multidut_port_info0-cold-ixr-aaa-14|3] --------------------------------------------------------------------------------------------------- live log setup ---------------------------------------------------------------------------------------------------- 13:46:03 __init__.set_default L0053 INFO | Completeness level not set during test execution. Setting to default level: CompletenessLevel.basic 13:46:03 __init__.check_test_completeness L0151 INFO | Test has no defined levels. Continue without test completeness checks 13:46:03 __init__.loganalyzer L0067 INFO | Log analyzer is disabled 13:46:03 __init__.store_fixture_values L0020 INFO | store memory_utilization test_pfc_pause_single_lossless_prio_reboot[multidut_port_info0-cold-ixr-aaa-14|3] 13:46:03 __init__._fixture_generator_decorator L0081 INFO | -------------------- fixture ignore_route_check_for_cisco_8000 setup starts -------------------- 13:46:03 __init__._fixture_generator_decorator L0085 INFO | -------------------- fixture ignore_route_check_for_cisco_8000 setup ends -------------------- 13:46:03 __init__._fixture_generator_decorator L0081 INFO | -------------------- fixture multidut_port_info setup starts -------------------- 13:46:03 __init__._fixture_generator_decorator L0085 INFO | -------------------- fixture multidut_port_info setup ends -------------------- 13:46:03 __init__._fixture_generator_decorator L0081 INFO | -------------------- fixture number_of_tx_rx_ports setup starts -------------------- 13:46:03 __init__._fixture_generator_decorator L0085 INFO | -------------------- fixture number_of_tx_rx_ports setup ends -------------------- 13:46:03 __init__._fixture_generator_decorator L0081 INFO | -------------------- fixture setup_ports_and_dut setup starts -------------------- 13:46:03 helper.setup_ports_and_dut L0111 INFO | Running test for testbed subtype: single-dut-multi-asic-aaa 13:46:04 __init__._fixture_generator_decorator L0085 INFO | -------------------- fixture setup_ports_and_dut setup ends -------------------- 13:46:04 __init__._fixture_generator_decorator L0081 INFO | -------------------- fixture reboot_duts_and_disable_wd setup starts -------------------- 13:46:05 sonic.get_asic_name L1797 INFO | asic: unknown 13:46:05 helper.skip_warm_reboot L0045 INFO | Reboot type cold is supported on broadcom switches 13:46:05 sonic.get_asic_name L1797 INFO | asic: unknown 13:46:05 helper.skip_warm_reboot L0045 INFO | Reboot type cold is supported on broadcom switches 13:46:08 helper.save_config_and_reboot L0210 INFO | Issuing a cold reboot on the dut ixr-aaa-14 13:46:11 reboot.reboot L0271 INFO | Reboot ixr-aaa-14: wait[120], timeout[300] 13:46:11 reboot.reboot L0273 INFO | DUT ixr-aaa-14 create a file /dev/shm/test_reboot before rebooting 13:46:12 reboot.collect_console_log L0566 INFO | start: collect console log 13:46:12 dut_utils.creds_on_dut L0471 INFO | dut ixr-aaa-14 belongs to groups ['ixia-sonic', 'sonic', 'sonic_aaa', 'fanout'] 13:46:12 dut_utils.creds_on_dut L0496 INFO | skip empty var file /data/sonic-mgmt/tests/common/helpers/../../../ansible/group_vars/all/corefile_uploader.yml 13:46:12 transport._log L1873 INFO | Connected (version 2.0, client OpenSSH_9.2p1) 13:46:16 transport._log L1873 INFO | Auth banner: b'Debian GNU/Linux 12 \\n \\l\n\n' 13:46:16 transport._log L1873 INFO | Authentication (password) failed. 13:46:16 transport._log L1873 INFO | Connected (version 2.0, client OpenSSH_9.2p1) 13:46:16 transport._log L1873 INFO | Auth banner: b'Debian GNU/Linux 12 \\n \\l\n\n' 13:46:16 transport._log L1873 INFO | Authentication (password) successful! 13:46:17 reboot.try_create_dut_console L0559 WARNING| Fail to create dut console. Please check console config or if console works ro not. 'ManagementIp' 13:46:17 reboot.collect_console_log L0576 WARNING| dut console is not ready, we cannot get log by console 13:46:17 reboot.wait_for_shutdown L0164 INFO | waiting for ssh to drop on ixr-aaa-14 13:46:17 reboot.execute_reboot_command L0205 INFO | rebooting ixr-aaa-14 with command "reboot" 13:46:35 reboot.wait_for_startup L0185 INFO | waiting for ssh to startup on ixr-aaa-14 13:48:05 reboot.wait_for_startup L0197 INFO | ssh has started up on ixr-aaa-14 13:48:05 reboot.reboot L0300 INFO | waiting for switch ixr-aaa-14 to initialize 13:52:31 processes_utils.wait_critical_processes L0078 INFO | Wait until all critical processes are healthy in 600 sec 13:52:31 processes_utils._all_critical_processes_ L0045 INFO | Check critical processes status 13:52:40 reboot.reboot L0330 INFO | cold reboot finished on ixr-aaa-14 13:52:41 reboot.reboot L0333 INFO | DUT ixr-aaa-14 up since 2025-05-01 13:47:31.410000 13:52:41 helper.save_config_and_reboot L0214 INFO | Wait until the system is stable 13:52:47 sonic_asic.check_bgp_session_state L0714 INFO | bgp neighbors that match the state: ['3.3.3.2', '3333::3:2'] on namespace asic0 13:52:47 sonic_asic.check_bgp_session_state L0715 INFO | bgp neighbors to be checked on the state: [] on namespace asic0 13:52:48 sonic_asic.check_bgp_session_state L0714 INFO | bgp neighbors that match the state: ['3.3.3.1', '3333::3:1'] on namespace asic1 13:52:48 sonic_asic.check_bgp_session_state L0715 INFO | bgp neighbors to be checked on the state: [] on namespace asic1 13:52:48 parallel.on_terminate L0085 INFO | process save_config_and_reboot--<MultiAsicSonicHost ixr-aaa-14> terminated with exit code None 13:52:48 parallel.parallel_run L0221 INFO | Completed running processes for target "save_config_and_reboot" in 0:06:43.000484 seconds 13:52:54 __init__._fixture_generator_decorator L0085 INFO | -------------------- fixture reboot_duts_and_disable_wd setup ends -------------------- 13:52:54 __init__.pytest_runtest_setup L0034 INFO | collect memory before test test_pfc_pause_single_lossless_prio_reboot[multidut_port_info0-cold-ixr-aaa-14|3] 13:52:54 __init__.pytest_runtest_setup L0054 INFO | Before test: collected memory_values {'before_test': {}, 'after_test': {}} -------------------------- curtailed output ----------------------- 13:57:03 traffic_generation.run_traffic L0465 INFO | In-flight traffic statistics for flows: ['Test Flow Prio 3', 'Background Flow Prio 1', 'Background Flow Prio 4', 'Background Flow Prio 5', 'Background Flow Prio 6', 'Background Flow Prio 2', 'Background Flow Prio 0'] 13:57:03 traffic_generation.run_traffic L0466 INFO | In-flight TX frames: [70428, 63697318, 63697318, 63697318, 63697318, 63697318, 63697318] 13:57:03 traffic_generation.run_traffic L0467 INFO | In-flight RX frames: [0, 63697318, 63697318, 63697318, 63697318, 63697318, 63697318] 13:57:21 traffic_generation.run_traffic L0468 INFO | DUT polling complete 13:57:21 traffic_generation.run_traffic L0479 INFO | Checking if all flows have stopped. Attempt sonic-net#1 13:57:22 traffic_generation.run_traffic L0486 INFO | All test and background traffic flows stopped 13:57:24 traffic_generation.run_traffic L0514 INFO | Dumping per-flow statistics 13:57:25 traffic_generation.run_traffic L0516 INFO | Stopping transmit on all remaining flows 13:57:32 snappi_api.info L1132 INFO | Flows stop 6.258s PASSED -------------------------- curtailed output ----------------------- snappi_tests/pfc/test_pfc_pause_lossless_with_snappi.py::test_pfc_pause_multi_lossless_prio_reboot[multidut_port_info0-cold] --------------------------------------------------------------------------------------------------- live log setup ---------------------------------------------------------------------------------------------------- 14:17:56 __init__.set_default L0053 INFO | Completeness level not set during test execution. Setting to default level: CompletenessLevel.basic 14:17:56 __init__.check_test_completeness L0151 INFO | Test has no defined levels. Continue without test completeness checks 14:17:56 __init__.loganalyzer L0067 INFO | Log analyzer is disabled 14:17:56 __init__.store_fixture_values L0020 INFO | store memory_utilization test_pfc_pause_multi_lossless_prio_reboot[multidut_port_info0-cold] 14:17:56 __init__._fixture_generator_decorator L0081 INFO | -------------------- fixture ignore_route_check_for_cisco_8000 setup starts -------------------- 14:17:56 __init__._fixture_generator_decorator L0085 INFO | -------------------- fixture ignore_route_check_for_cisco_8000 setup ends -------------------- 14:17:56 __init__._fixture_generator_decorator L0081 INFO | -------------------- fixture multidut_port_info setup starts -------------------- 14:17:56 __init__._fixture_generator_decorator L0085 INFO | -------------------- fixture multidut_port_info setup ends -------------------- 14:17:56 __init__._fixture_generator_decorator L0081 INFO | -------------------- fixture number_of_tx_rx_ports setup starts -------------------- 14:17:56 __init__._fixture_generator_decorator L0085 INFO | -------------------- fixture number_of_tx_rx_ports setup ends -------------------- 14:17:56 __init__._fixture_generator_decorator L0081 INFO | -------------------- fixture setup_ports_and_dut setup starts -------------------- 14:17:56 helper.setup_ports_and_dut L0111 INFO | Running test for testbed subtype: single-dut-multi-asic-aaa 14:17:57 __init__._fixture_generator_decorator L0085 INFO | -------------------- fixture setup_ports_and_dut setup ends -------------------- 14:17:57 __init__._fixture_generator_decorator L0081 INFO | -------------------- fixture reboot_duts_and_disable_wd setup starts -------------------- 14:17:58 sonic.get_asic_name L1797 INFO | asic: unknown 14:17:58 helper.skip_warm_reboot L0045 INFO | Reboot type cold is supported on broadcom switches 14:17:58 sonic.get_asic_name L1797 INFO | asic: unknown 14:17:58 helper.skip_warm_reboot L0045 INFO | Reboot type cold is supported on broadcom switches 14:18:01 helper.save_config_and_reboot L0210 INFO | Issuing a cold reboot on the dut ixr-aaa-14 14:18:04 reboot.reboot L0271 INFO | Reboot ixr-aaa-14: wait[120], timeout[300] 14:18:04 reboot.reboot L0273 INFO | DUT ixr-aaa-14 create a file /dev/shm/test_reboot before rebooting 14:18:05 reboot.collect_console_log L0566 INFO | start: collect console log 14:18:05 dut_utils.creds_on_dut L0471 INFO | dut ixr-aaa-14 belongs to groups ['ixia-sonic', 'sonic', 'sonic_aaa', 'fanout'] 14:18:05 dut_utils.creds_on_dut L0496 INFO | skip empty var file /data/sonic-mgmt/tests/common/helpers/../../../ansible/group_vars/all/corefile_uploader.yml 14:18:05 transport._log L1873 INFO | Connected (version 2.0, client OpenSSH_9.2p1) 14:18:07 transport._log L1873 INFO | Auth banner: b'Debian GNU/Linux 12 \\n \\l\n\n' 14:18:07 transport._log L1873 INFO | Authentication (password) failed. 14:18:07 transport._log L1873 INFO | Connected (version 2.0, client OpenSSH_9.2p1) 14:18:07 transport._log L1873 INFO | Auth banner: b'Debian GNU/Linux 12 \\n \\l\n\n' 14:18:07 transport._log L1873 INFO | Authentication (password) successful! 14:18:08 reboot.try_create_dut_console L0559 WARNING| Fail to create dut console. Please check console config or if console works ro not. 'ManagementIp' 14:18:08 reboot.collect_console_log L0576 WARNING| dut console is not ready, we cannot get log by console 14:18:10 reboot.wait_for_shutdown L0164 INFO | waiting for ssh to drop on ixr-aaa-14 14:18:10 reboot.execute_reboot_command L0205 INFO | rebooting ixr-aaa-14 with command "reboot" 14:18:28 reboot.wait_for_startup L0185 INFO | waiting for ssh to startup on ixr-aaa-14 14:19:56 reboot.wait_for_startup L0197 INFO | ssh has started up on ixr-aaa-14 14:19:56 reboot.reboot L0300 INFO | waiting for switch ixr-aaa-14 to initialize 14:24:22 processes_utils.wait_critical_processes L0078 INFO | Wait until all critical processes are healthy in 600 sec 14:24:22 processes_utils._all_critical_processes_ L0045 INFO | Check critical processes status 14:24:30 reboot.reboot L0330 INFO | cold reboot finished on ixr-aaa-14 14:24:31 reboot.reboot L0333 INFO | DUT ixr-aaa-14 up since 2025-05-01 14:19:23.320000 14:24:31 helper.save_config_and_reboot L0214 INFO | Wait until the system is stable 14:24:37 sonic_asic.check_bgp_session_state L0714 INFO | bgp neighbors that match the state: ['3.3.3.2', '3333::3:2'] on namespace asic0 14:24:37 sonic_asic.check_bgp_session_state L0715 INFO | bgp neighbors to be checked on the state: [] on namespace asic0 14:24:38 sonic_asic.check_bgp_session_state L0714 INFO | bgp neighbors that match the state: ['3.3.3.1', '3333::3:1'] on namespace asic1 14:24:38 sonic_asic.check_bgp_session_state L0715 INFO | bgp neighbors to be checked on the state: [] on namespace asic1 14:24:38 parallel.on_terminate L0085 INFO | process save_config_and_reboot--<MultiAsicSonicHost ixr-aaa-14> terminated with exit code None 14:24:38 parallel.parallel_run L0221 INFO | Completed running processes for target "save_config_and_reboot" in 0:06:39.645613 seconds 14:24:44 __init__._fixture_generator_decorator L0085 INFO | -------------------- fixture reboot_duts_and_disable_wd setup ends -------------------- 14:24:44 __init__.pytest_runtest_setup L0034 INFO | collect memory before test test_pfc_pause_multi_lossless_prio_reboot[multidut_port_info0-cold] 14:24:44 __init__.pytest_runtest_setup L0054 INFO | Before test: collected memory_values {'before_test': {}, 'after_test': {}} -------------------------- curtailed output ----------------------- 14:28:16 traffic_generation.run_traffic L0465 INFO | In-flight traffic statistics for flows: ['Test Flow Prio 3', 'Test Flow Prio 4', 'Background Flow Prio 1', 'Background Flow Prio 5', 'Background Flow Prio 6', 'Background Flow Prio 2', 'Background Flow Prio 0'] 14:28:16 traffic_generation.run_traffic L0466 INFO | In-flight TX frames: [68215, 68215, 116379310, 116379310, 116379310, 116379310, 116379310] 14:28:16 traffic_generation.run_traffic L0467 INFO | In-flight RX frames: [0, 0, 116379310, 116379310, 116379310, 116379310, 116379310] 14:28:49 traffic_generation.run_traffic L0468 INFO | DUT polling complete 14:28:49 traffic_generation.run_traffic L0479 INFO | Checking if all flows have stopped. Attempt sonic-net#1 14:28:50 traffic_generation.run_traffic L0486 INFO | All test and background traffic flows stopped 14:28:52 traffic_generation.run_traffic L0514 INFO | Dumping per-flow statistics 14:28:53 traffic_generation.run_traffic L0516 INFO | Stopping transmit on all remaining flows 14:29:00 snappi_api.info L1132 INFO | Flows stop 6.248s PASSED T2 Verification: AzDevOps@7c3002ea4994:/data/tests$ date;python3 -m pytest --inventory ../ansible/ixia-sonic --host-pattern board73,board74 --testbed ixre-chassis17-t2 --testbed_file ../ansible/testbed.csv --show-capture=stdout --log-cli-level info --showlocals -ra --allow_recover --junit-xml=/tmp/pfc/p.xml --skip_sanity --log-file=/tmp/pfc/p.log snappi_tests/pfc/test_pfc_pause_lossless_with_snappi.py -k reboot --pdb Wed Apr 30 20:44:41 UTC 2025 --------------- curtailed output ---------------- snappi_tests/pfc/test_pfc_pause_lossless_with_snappi.py::test_pfc_pause_single_lossless_prio_reboot[multidut_port_info0-cold-board71|3] --------------------------------------------------------------------------------------------------- live log setup ---------------------------------------------------------------------------------------------------- 20:45:40 __init__.set_default L0053 INFO | Completeness level not set during test execution. Setting to default level: CompletenessLevel.basic 20:45:40 __init__.check_test_completeness L0151 INFO | Test has no defined levels. Continue without test completeness checks 20:45:40 __init__.loganalyzer L0067 INFO | Log analyzer is disabled 20:45:40 __init__._fixture_generator_decorator L0081 INFO | -------------------- fixture ignore_route_check_for_cisco_8000 setup starts -------------------- 20:45:40 __init__._fixture_generator_decorator L0085 INFO | -------------------- fixture ignore_route_check_for_cisco_8000 setup ends -------------------- 20:45:40 __init__._fixture_generator_decorator L0081 INFO | -------------------- fixture multidut_port_info setup starts -------------------- 20:45:40 __init__._fixture_generator_decorator L0085 INFO | -------------------- fixture multidut_port_info setup ends -------------------- 20:45:40 __init__._fixture_generator_decorator L0081 INFO | -------------------- fixture number_of_tx_rx_ports setup starts -------------------- 20:45:40 __init__._fixture_generator_decorator L0085 INFO | -------------------- fixture number_of_tx_rx_ports setup ends -------------------- 20:45:40 __init__._fixture_generator_decorator L0081 INFO | -------------------- fixture setup_ports_and_dut setup starts -------------------- 20:45:40 helper.setup_ports_and_dut L0115 INFO | Running test for testbed subtype: multi-dut-long-to-short-link 20:45:43 snappi_fixtures.__intf_config_multidut L0858 INFO | Configuring Dut: board73 with port Ethernet16 with IP 20.1.1.2/31 20:45:43 snappi_fixtures.__intf_config_multidut L0878 INFO | Adding ip:20.1.1.2 on port:Ethernet16 on DUT:board73 20:45:44 snappi_fixtures.__intf_config_multidut L0858 INFO | Configuring Dut: board74 with port Ethernet0 with IP 20.1.1.0/31 20:45:44 snappi_fixtures.__intf_config_multidut L0878 INFO | Adding ip:20.1.1.0 on port:Ethernet0 on DUT:board74 20:45:45 __init__._fixture_generator_decorator L0085 INFO | -------------------- fixture setup_ports_and_dut setup ends -------------------- 20:45:45 __init__._fixture_generator_decorator L0081 INFO | -------------------- fixture reboot_duts_and_disable_wd setup starts -------------------- 20:45:46 sonic.get_asic_name L1778 INFO | asic: unknown 20:45:46 helper.skip_warm_reboot L0049 INFO | Reboot type cold is supported on broadcom switches 20:45:46 sonic.get_asic_name L1778 INFO | asic: unknown 20:45:46 helper.skip_warm_reboot L0049 INFO | Reboot type cold is supported on broadcom switches 20:45:49 helper.save_config_and_reboot L0408 INFO | Issuing a cold reboot on the dut board74 20:45:49 helper.save_config_and_reboot L0408 INFO | Issuing a cold reboot on the dut board73 20:45:52 reboot.reboot L0269 INFO | Reboot board74: wait[900], timeout[600] 20:45:52 reboot.reboot L0271 INFO | DUT board74 create a file /dev/shm/test_reboot before rebooting 20:45:52 reboot.reboot L0269 INFO | Reboot board73: wait[900], timeout[600] 20:45:52 reboot.reboot L0271 INFO | DUT board73 create a file /dev/shm/test_reboot before rebooting 20:45:54 reboot.wait_for_shutdown L0162 INFO | waiting for ssh to drop on board74 20:45:54 reboot.execute_reboot_command L0202 INFO | rebooting board74 with command "reboot" 20:45:54 reboot.wait_for_shutdown L0162 INFO | waiting for ssh to drop on board73 20:45:54 reboot.execute_reboot_command L0202 INFO | rebooting board73 with command "reboot" 20:46:15 reboot.wait_for_startup L0183 INFO | waiting for ssh to startup on board74 20:46:16 reboot.wait_for_startup L0183 INFO | waiting for ssh to startup on board73 20:48:38 reboot.wait_for_startup L0194 INFO | ssh has started up on board73 20:48:38 reboot.reboot L0288 INFO | waiting for switch board73 to initialize 20:48:40 reboot.wait_for_startup L0194 INFO | ssh has started up on board74 20:48:40 reboot.reboot L0288 INFO | waiting for switch board74 to initialize 20:52:24 processes_utils.wait_critical_processes L0079 INFO | Wait until all critical processes are healthy in 600 sec 20:52:24 processes_utils._all_critical_processes_ L0045 INFO | Check critical processes status 20:52:25 processes_utils.wait_critical_processes L0079 INFO | Wait until all critical processes are healthy in 600 sec 20:52:25 processes_utils._all_critical_processes_ L0045 INFO | Check critical processes status 20:52:32 processes_utils.get_critical_processes_s L0035 INFO | The status of checking process in container 'lldp0' is: False 20:52:32 processes_utils.get_critical_processes_s L0037 INFO | The processes not running in container 'lldp0' are: '['lldp-syncd', 'lldpmgrd']' 20:52:32 processes_utils.get_critical_processes_s L0035 INFO | The status of checking process in container 'lldp0' is: False 20:52:32 processes_utils.get_critical_processes_s L0037 INFO | The processes not running in container 'lldp0' are: '['lldp-syncd', 'lldpmgrd']' 20:52:52 processes_utils._all_critical_processes_ L0045 INFO | Check critical processes status 20:52:52 processes_utils._all_critical_processes_ L0045 INFO | Check critical processes status 20:52:59 processes_utils.get_critical_processes_s L0035 INFO | The status of checking process in container 'lldp0' is: False 20:52:59 processes_utils.get_critical_processes_s L0037 INFO | The processes not running in container 'lldp0' are: '['lldp-syncd', 'lldpmgrd']' 20:53:00 processes_utils.get_critical_processes_s L0035 INFO | The status of checking process in container 'lldp0' is: False 20:53:00 processes_utils.get_critical_processes_s L0037 INFO | The processes not running in container 'lldp0' are: '['lldp-syncd', 'lldpmgrd']' 20:53:20 processes_utils._all_critical_processes_ L0045 INFO | Check critical processes status 20:53:20 processes_utils._all_critical_processes_ L0045 INFO | Check critical processes status 20:53:27 reboot.reboot L0318 INFO | cold reboot finished on board73 20:53:28 reboot.reboot L0318 INFO | cold reboot finished on board74 20:53:28 reboot.reboot L0321 INFO | DUT board73 up since 2025-04-30 20:47:22.580000 20:53:28 reboot.reboot L0321 INFO | DUT board74 up since 2025-04-30 20:47:19.270000 20:53:29 helper.save_config_and_reboot L0412 INFO | Wait until the system is stable 20:53:29 helper.save_config_and_reboot L0412 INFO | Wait until the system is stable 20:53:35 sonic_asic.check_bgp_session_state L0692 INFO | bgp neighbors that match the state: ['3.3.3.1', '3.3.3.2', '3.3.3.6', '3.3.3.8', '3.3.3.12', '3.3.3.14', '3.3.3.20', '3333::3:1', '3333::3:2', '3333::3:6', '3333::3:8', '3333::3:12', '3333::3:14', '3333::3:20'] on namespace asic0 20:53:35 sonic_asic.check_bgp_session_state L0693 INFO | bgp neighbors to be checked on the state: [] on namespace asic0 20:53:35 sonic_asic.check_bgp_session_state L0692 INFO | bgp neighbors that match the state: ['3.3.3.1', '3.3.3.2', '3.3.3.6', '3.3.3.8', '3.3.3.14', '3.3.3.18', '3.3.3.20', '3333::3:1', '3333::3:2', '3333::3:6', '3333::3:8', '3333::3:14', '3333::3:18', '3333::3:20'] on namespace asic0 20:53:35 sonic_asic.check_bgp_session_state L0693 INFO | bgp neighbors to be checked on the state: [] on namespace asic0 20:53:36 sonic_asic.check_bgp_session_state L0692 INFO | bgp neighbors that match the state: ['3.3.3.1', '3.3.3.2', '3.3.3.6', '3.3.3.8', '3.3.3.12', '3.3.3.14', '3.3.3.18', '3333::3:1', '3333::3:2', '3333::3:6', '3333::3:8', '3333::3:12', '3333::3:14', '3333::3:18'] on namespace asic1 20:53:36 sonic_asic.check_bgp_session_state L0693 INFO | bgp neighbors to be checked on the state: [] on namespace asic1 20:53:36 parallel.on_terminate L0085 INFO | process save_config_and_reboot--<MultiAsicSonicHost board74> terminated with exit code None 20:53:36 sonic_asic.check_bgp_session_state L0692 INFO | bgp neighbors that match the state: ['3.3.3.1', '3.3.3.2', '3.3.3.6', '3.3.3.8', '3.3.3.12', '3.3.3.18', '3.3.3.20', '3333::3:1', '3333::3:2', '3333::3:6', '3333::3:8', '3333::3:12', '3333::3:18', '3333::3:20'] on namespace asic1 20:53:36 sonic_asic.check_bgp_session_state L0693 INFO | bgp neighbors to be checked on the state: [] on namespace asic1 20:53:36 parallel.on_terminate L0085 INFO | process save_config_and_reboot--<MultiAsicSonicHost board73> terminated with exit code None 20:53:36 parallel.parallel_run L0221 INFO | Completed running processes for target "save_config_and_reboot" in 0:07:49.938416 seconds 20:53:50 __init__._fixture_generator_decorator L0085 INFO | -------------------- fixture reboot_duts_and_disable_wd setup ends -------------------- --------------- curtailed output ---------------- PASSED [ 8%] -------------------------------------------------------------------------------------------------- live log teardown -------------------------------------------------------------------------------------------------- 20:57:53 __init__._fixture_generator_decorator L0093 INFO | -------------------- fixture reboot_duts_and_disable_wd teardown starts -------------------- 20:58:06 config_reload.config_reload L0145 INFO | reloading config_db 21:02:39 processes_utils.wait_critical_processes L0079 INFO | Wait until all critical processes are healthy in 600 sec 21:02:39 processes_utils._all_critical_processes_ L0045 INFO | Check critical processes status 21:02:46 processes_utils.get_critical_processes_s L0035 INFO | The status of checking process in container 'lldp0' is: False 21:02:46 processes_utils.get_critical_processes_s L0037 INFO | The processes not running in container 'lldp0' are: '['lldp-syncd', 'lldpmgrd']' 21:03:06 processes_utils._all_critical_processes_ L0045 INFO | Check critical processes status 21:03:14 config_reload.config_reload L0145 INFO | reloading config_db 21:07:44 processes_utils.wait_critical_processes L0079 INFO | Wait until all critical processes are healthy in 600 sec 21:07:44 processes_utils._all_critical_processes_ L0045 INFO | Check critical processes status 21:07:52 processes_utils.get_critical_processes_s L0035 INFO | The status of checking process in container 'lldp0' is: False 21:07:52 processes_utils.get_critical_processes_s L0037 INFO | The processes not running in container 'lldp0' are: '['lldp-syncd', 'lldpmgrd']' 21:08:12 processes_utils._all_critical_processes_ L0045 INFO | Check critical processes status 21:08:19 __init__._fixture_generator_decorator L0102 INFO | -------------------- fixture reboot_duts_and_disable_wd teardown ends -------------------- 21:08:19 __init__._fixture_generator_decorator L0093 INFO | -------------------- fixture setup_ports_and_dut teardown starts -------------------- 21:08:22 snappi_fixtures.__intf_config_multidut L0858 INFO | Configuring Dut: board73 with port Ethernet16 with IP 20.1.1.2/31 21:08:22 snappi_fixtures.__intf_config_multidut L0880 INFO | Removing ip:20.1.1.2 on port:Ethernet16 on DUT:board73 21:08:23 snappi_fixtures.__intf_config_multidut L0858 INFO | Configuring Dut: board74 with port Ethernet0 with IP 20.1.1.0/31 21:08:23 snappi_fixtures.__intf_config_multidut L0880 INFO | Removing ip:20.1.1.0 on port:Ethernet0 on DUT:board74 21:08:24 __init__._fixture_generator_decorator L0102 INFO | -------------------- fixture setup_ports_and_dut teardown ends -------------------- 21:08:24 __init__._fixture_generator_decorator L0093 INFO | -------------------- fixture number_of_tx_rx_ports teardown starts -------------------- 21:08:24 __init__._fixture_generator_decorator L0102 INFO | -------------------- fixture number_of_tx_rx_ports teardown ends -------------------- 21:08:24 __init__._fixture_generator_decorator L0093 INFO | -------------------- fixture multidut_port_info teardown starts -------------------- 21:08:24 __init__._fixture_generator_decorator L0102 INFO | -------------------- fixture multidut_port_info teardown ends -------------------- 21:08:24 __init__._fixture_generator_decorator L0093 INFO | -------------------- fixture ignore_route_check_for_cisco_8000 teardown starts -------------------- 21:08:24 __init__._fixture_generator_decorator L0102 INFO | -------------------- fixture ignore_route_check_for_cisco_8000 teardown ends -------------------- --------------- curtailed output ---------------- ======================================================================= 8 passed, 16 skipped, 20 deselected, 71 warnings in 10116.17s (2:48:36) ======================================================================= INFO:root:Can not get Allure report URL. Please check logs co-authorized by: jianquanye@microsoft.com Signed-off-by: Guy Shemesh <gshemesh@nvidia.com>
gshemesh2
pushed a commit
to gshemesh2/sonic-mgmt
that referenced
this pull request
Jan 26, 2026
…in pfc_pause_lossless test (sonic-net#18105) Description of PR In tests/snappi_tests/pfc/test_pfc_pause_lossless_with_snappi.py, we have reboot testcases. We disable the PFCWD and Credit-WD in the testcase and reboot the the DUT depending upon type of reboot. However, on cold-reboot, the Credit-WD is getting reset back. This causes the lossless test flows on IXIA to continuously send traffic despite of the pause-storm. The run_traffic in traffic_generation.py is trying to determine if the test stops within a stipulated duration, which it does not and throws an assert that flows have not stopped within the given time duration and attempts. Summary: Fixes # (issue) sonic-net#18082 Type of change Bug fix Testbed and Framework(new/improvement) New Test case Skipped for non-supported platforms Test case improvement Back port request 202012 202205 202305 202311 202405 202411 Approach What is the motivation for this PR? The reboot_dut was called after the PFCWD and Credit-WD disable. The reboot caused the Credit-WD to reset back and hence test failed. How did you do it? Created a single fixture called reboot_duts_and_disable_wd to reboot the DUTs and disable Credit and PFC watch-dogs. This fixture is added in snappi_tests/files/helper.py. Modified the testcase - snappi_tests/pfc/test_pfc_pause_lossless_with_snappi.py to incorporate the above fixture. How did you verify/test it? T0/T1 setup verification: AzDevOps@sonic_mgmt_nokia_202411:/data/sonic-mgmt/tests$ date;python3 -m pytest --inventory ../ansible/ixia-sonic --host-pattern ixr-x3b-14 --testbed ixr-aaa-14-t2 --testbed_file ../ansible/testbed.csv --log-cli-level info --log-file-level info --kube_master unset --showlocals -ra --show-capture stdout --junit-xml=/tmp/p.xml --skip_sanity --log-file=/tmp/p.log --cache-clear --topology tgen snappi_tests/pfc/test_pfc_pause_lossless_with_snappi.py -k reboot Thu May 1 13:45:24 UTC 2025 -------------------------- curtailed output ----------------------- snappi_tests/pfc/test_pfc_pause_lossless_with_snappi.py::test_pfc_pause_single_lossless_prio_reboot[multidut_port_info0-cold-ixr-aaa-14|3] --------------------------------------------------------------------------------------------------- live log setup ---------------------------------------------------------------------------------------------------- 13:46:03 __init__.set_default L0053 INFO | Completeness level not set during test execution. Setting to default level: CompletenessLevel.basic 13:46:03 __init__.check_test_completeness L0151 INFO | Test has no defined levels. Continue without test completeness checks 13:46:03 __init__.loganalyzer L0067 INFO | Log analyzer is disabled 13:46:03 __init__.store_fixture_values L0020 INFO | store memory_utilization test_pfc_pause_single_lossless_prio_reboot[multidut_port_info0-cold-ixr-aaa-14|3] 13:46:03 __init__._fixture_generator_decorator L0081 INFO | -------------------- fixture ignore_route_check_for_cisco_8000 setup starts -------------------- 13:46:03 __init__._fixture_generator_decorator L0085 INFO | -------------------- fixture ignore_route_check_for_cisco_8000 setup ends -------------------- 13:46:03 __init__._fixture_generator_decorator L0081 INFO | -------------------- fixture multidut_port_info setup starts -------------------- 13:46:03 __init__._fixture_generator_decorator L0085 INFO | -------------------- fixture multidut_port_info setup ends -------------------- 13:46:03 __init__._fixture_generator_decorator L0081 INFO | -------------------- fixture number_of_tx_rx_ports setup starts -------------------- 13:46:03 __init__._fixture_generator_decorator L0085 INFO | -------------------- fixture number_of_tx_rx_ports setup ends -------------------- 13:46:03 __init__._fixture_generator_decorator L0081 INFO | -------------------- fixture setup_ports_and_dut setup starts -------------------- 13:46:03 helper.setup_ports_and_dut L0111 INFO | Running test for testbed subtype: single-dut-multi-asic-aaa 13:46:04 __init__._fixture_generator_decorator L0085 INFO | -------------------- fixture setup_ports_and_dut setup ends -------------------- 13:46:04 __init__._fixture_generator_decorator L0081 INFO | -------------------- fixture reboot_duts_and_disable_wd setup starts -------------------- 13:46:05 sonic.get_asic_name L1797 INFO | asic: unknown 13:46:05 helper.skip_warm_reboot L0045 INFO | Reboot type cold is supported on broadcom switches 13:46:05 sonic.get_asic_name L1797 INFO | asic: unknown 13:46:05 helper.skip_warm_reboot L0045 INFO | Reboot type cold is supported on broadcom switches 13:46:08 helper.save_config_and_reboot L0210 INFO | Issuing a cold reboot on the dut ixr-aaa-14 13:46:11 reboot.reboot L0271 INFO | Reboot ixr-aaa-14: wait[120], timeout[300] 13:46:11 reboot.reboot L0273 INFO | DUT ixr-aaa-14 create a file /dev/shm/test_reboot before rebooting 13:46:12 reboot.collect_console_log L0566 INFO | start: collect console log 13:46:12 dut_utils.creds_on_dut L0471 INFO | dut ixr-aaa-14 belongs to groups ['ixia-sonic', 'sonic', 'sonic_aaa', 'fanout'] 13:46:12 dut_utils.creds_on_dut L0496 INFO | skip empty var file /data/sonic-mgmt/tests/common/helpers/../../../ansible/group_vars/all/corefile_uploader.yml 13:46:12 transport._log L1873 INFO | Connected (version 2.0, client OpenSSH_9.2p1) 13:46:16 transport._log L1873 INFO | Auth banner: b'Debian GNU/Linux 12 \\n \\l\n\n' 13:46:16 transport._log L1873 INFO | Authentication (password) failed. 13:46:16 transport._log L1873 INFO | Connected (version 2.0, client OpenSSH_9.2p1) 13:46:16 transport._log L1873 INFO | Auth banner: b'Debian GNU/Linux 12 \\n \\l\n\n' 13:46:16 transport._log L1873 INFO | Authentication (password) successful! 13:46:17 reboot.try_create_dut_console L0559 WARNING| Fail to create dut console. Please check console config or if console works ro not. 'ManagementIp' 13:46:17 reboot.collect_console_log L0576 WARNING| dut console is not ready, we cannot get log by console 13:46:17 reboot.wait_for_shutdown L0164 INFO | waiting for ssh to drop on ixr-aaa-14 13:46:17 reboot.execute_reboot_command L0205 INFO | rebooting ixr-aaa-14 with command "reboot" 13:46:35 reboot.wait_for_startup L0185 INFO | waiting for ssh to startup on ixr-aaa-14 13:48:05 reboot.wait_for_startup L0197 INFO | ssh has started up on ixr-aaa-14 13:48:05 reboot.reboot L0300 INFO | waiting for switch ixr-aaa-14 to initialize 13:52:31 processes_utils.wait_critical_processes L0078 INFO | Wait until all critical processes are healthy in 600 sec 13:52:31 processes_utils._all_critical_processes_ L0045 INFO | Check critical processes status 13:52:40 reboot.reboot L0330 INFO | cold reboot finished on ixr-aaa-14 13:52:41 reboot.reboot L0333 INFO | DUT ixr-aaa-14 up since 2025-05-01 13:47:31.410000 13:52:41 helper.save_config_and_reboot L0214 INFO | Wait until the system is stable 13:52:47 sonic_asic.check_bgp_session_state L0714 INFO | bgp neighbors that match the state: ['3.3.3.2', '3333::3:2'] on namespace asic0 13:52:47 sonic_asic.check_bgp_session_state L0715 INFO | bgp neighbors to be checked on the state: [] on namespace asic0 13:52:48 sonic_asic.check_bgp_session_state L0714 INFO | bgp neighbors that match the state: ['3.3.3.1', '3333::3:1'] on namespace asic1 13:52:48 sonic_asic.check_bgp_session_state L0715 INFO | bgp neighbors to be checked on the state: [] on namespace asic1 13:52:48 parallel.on_terminate L0085 INFO | process save_config_and_reboot--<MultiAsicSonicHost ixr-aaa-14> terminated with exit code None 13:52:48 parallel.parallel_run L0221 INFO | Completed running processes for target "save_config_and_reboot" in 0:06:43.000484 seconds 13:52:54 __init__._fixture_generator_decorator L0085 INFO | -------------------- fixture reboot_duts_and_disable_wd setup ends -------------------- 13:52:54 __init__.pytest_runtest_setup L0034 INFO | collect memory before test test_pfc_pause_single_lossless_prio_reboot[multidut_port_info0-cold-ixr-aaa-14|3] 13:52:54 __init__.pytest_runtest_setup L0054 INFO | Before test: collected memory_values {'before_test': {}, 'after_test': {}} -------------------------- curtailed output ----------------------- 13:57:03 traffic_generation.run_traffic L0465 INFO | In-flight traffic statistics for flows: ['Test Flow Prio 3', 'Background Flow Prio 1', 'Background Flow Prio 4', 'Background Flow Prio 5', 'Background Flow Prio 6', 'Background Flow Prio 2', 'Background Flow Prio 0'] 13:57:03 traffic_generation.run_traffic L0466 INFO | In-flight TX frames: [70428, 63697318, 63697318, 63697318, 63697318, 63697318, 63697318] 13:57:03 traffic_generation.run_traffic L0467 INFO | In-flight RX frames: [0, 63697318, 63697318, 63697318, 63697318, 63697318, 63697318] 13:57:21 traffic_generation.run_traffic L0468 INFO | DUT polling complete 13:57:21 traffic_generation.run_traffic L0479 INFO | Checking if all flows have stopped. Attempt sonic-net#1 13:57:22 traffic_generation.run_traffic L0486 INFO | All test and background traffic flows stopped 13:57:24 traffic_generation.run_traffic L0514 INFO | Dumping per-flow statistics 13:57:25 traffic_generation.run_traffic L0516 INFO | Stopping transmit on all remaining flows 13:57:32 snappi_api.info L1132 INFO | Flows stop 6.258s PASSED -------------------------- curtailed output ----------------------- snappi_tests/pfc/test_pfc_pause_lossless_with_snappi.py::test_pfc_pause_multi_lossless_prio_reboot[multidut_port_info0-cold] --------------------------------------------------------------------------------------------------- live log setup ---------------------------------------------------------------------------------------------------- 14:17:56 __init__.set_default L0053 INFO | Completeness level not set during test execution. Setting to default level: CompletenessLevel.basic 14:17:56 __init__.check_test_completeness L0151 INFO | Test has no defined levels. Continue without test completeness checks 14:17:56 __init__.loganalyzer L0067 INFO | Log analyzer is disabled 14:17:56 __init__.store_fixture_values L0020 INFO | store memory_utilization test_pfc_pause_multi_lossless_prio_reboot[multidut_port_info0-cold] 14:17:56 __init__._fixture_generator_decorator L0081 INFO | -------------------- fixture ignore_route_check_for_cisco_8000 setup starts -------------------- 14:17:56 __init__._fixture_generator_decorator L0085 INFO | -------------------- fixture ignore_route_check_for_cisco_8000 setup ends -------------------- 14:17:56 __init__._fixture_generator_decorator L0081 INFO | -------------------- fixture multidut_port_info setup starts -------------------- 14:17:56 __init__._fixture_generator_decorator L0085 INFO | -------------------- fixture multidut_port_info setup ends -------------------- 14:17:56 __init__._fixture_generator_decorator L0081 INFO | -------------------- fixture number_of_tx_rx_ports setup starts -------------------- 14:17:56 __init__._fixture_generator_decorator L0085 INFO | -------------------- fixture number_of_tx_rx_ports setup ends -------------------- 14:17:56 __init__._fixture_generator_decorator L0081 INFO | -------------------- fixture setup_ports_and_dut setup starts -------------------- 14:17:56 helper.setup_ports_and_dut L0111 INFO | Running test for testbed subtype: single-dut-multi-asic-aaa 14:17:57 __init__._fixture_generator_decorator L0085 INFO | -------------------- fixture setup_ports_and_dut setup ends -------------------- 14:17:57 __init__._fixture_generator_decorator L0081 INFO | -------------------- fixture reboot_duts_and_disable_wd setup starts -------------------- 14:17:58 sonic.get_asic_name L1797 INFO | asic: unknown 14:17:58 helper.skip_warm_reboot L0045 INFO | Reboot type cold is supported on broadcom switches 14:17:58 sonic.get_asic_name L1797 INFO | asic: unknown 14:17:58 helper.skip_warm_reboot L0045 INFO | Reboot type cold is supported on broadcom switches 14:18:01 helper.save_config_and_reboot L0210 INFO | Issuing a cold reboot on the dut ixr-aaa-14 14:18:04 reboot.reboot L0271 INFO | Reboot ixr-aaa-14: wait[120], timeout[300] 14:18:04 reboot.reboot L0273 INFO | DUT ixr-aaa-14 create a file /dev/shm/test_reboot before rebooting 14:18:05 reboot.collect_console_log L0566 INFO | start: collect console log 14:18:05 dut_utils.creds_on_dut L0471 INFO | dut ixr-aaa-14 belongs to groups ['ixia-sonic', 'sonic', 'sonic_aaa', 'fanout'] 14:18:05 dut_utils.creds_on_dut L0496 INFO | skip empty var file /data/sonic-mgmt/tests/common/helpers/../../../ansible/group_vars/all/corefile_uploader.yml 14:18:05 transport._log L1873 INFO | Connected (version 2.0, client OpenSSH_9.2p1) 14:18:07 transport._log L1873 INFO | Auth banner: b'Debian GNU/Linux 12 \\n \\l\n\n' 14:18:07 transport._log L1873 INFO | Authentication (password) failed. 14:18:07 transport._log L1873 INFO | Connected (version 2.0, client OpenSSH_9.2p1) 14:18:07 transport._log L1873 INFO | Auth banner: b'Debian GNU/Linux 12 \\n \\l\n\n' 14:18:07 transport._log L1873 INFO | Authentication (password) successful! 14:18:08 reboot.try_create_dut_console L0559 WARNING| Fail to create dut console. Please check console config or if console works ro not. 'ManagementIp' 14:18:08 reboot.collect_console_log L0576 WARNING| dut console is not ready, we cannot get log by console 14:18:10 reboot.wait_for_shutdown L0164 INFO | waiting for ssh to drop on ixr-aaa-14 14:18:10 reboot.execute_reboot_command L0205 INFO | rebooting ixr-aaa-14 with command "reboot" 14:18:28 reboot.wait_for_startup L0185 INFO | waiting for ssh to startup on ixr-aaa-14 14:19:56 reboot.wait_for_startup L0197 INFO | ssh has started up on ixr-aaa-14 14:19:56 reboot.reboot L0300 INFO | waiting for switch ixr-aaa-14 to initialize 14:24:22 processes_utils.wait_critical_processes L0078 INFO | Wait until all critical processes are healthy in 600 sec 14:24:22 processes_utils._all_critical_processes_ L0045 INFO | Check critical processes status 14:24:30 reboot.reboot L0330 INFO | cold reboot finished on ixr-aaa-14 14:24:31 reboot.reboot L0333 INFO | DUT ixr-aaa-14 up since 2025-05-01 14:19:23.320000 14:24:31 helper.save_config_and_reboot L0214 INFO | Wait until the system is stable 14:24:37 sonic_asic.check_bgp_session_state L0714 INFO | bgp neighbors that match the state: ['3.3.3.2', '3333::3:2'] on namespace asic0 14:24:37 sonic_asic.check_bgp_session_state L0715 INFO | bgp neighbors to be checked on the state: [] on namespace asic0 14:24:38 sonic_asic.check_bgp_session_state L0714 INFO | bgp neighbors that match the state: ['3.3.3.1', '3333::3:1'] on namespace asic1 14:24:38 sonic_asic.check_bgp_session_state L0715 INFO | bgp neighbors to be checked on the state: [] on namespace asic1 14:24:38 parallel.on_terminate L0085 INFO | process save_config_and_reboot--<MultiAsicSonicHost ixr-aaa-14> terminated with exit code None 14:24:38 parallel.parallel_run L0221 INFO | Completed running processes for target "save_config_and_reboot" in 0:06:39.645613 seconds 14:24:44 __init__._fixture_generator_decorator L0085 INFO | -------------------- fixture reboot_duts_and_disable_wd setup ends -------------------- 14:24:44 __init__.pytest_runtest_setup L0034 INFO | collect memory before test test_pfc_pause_multi_lossless_prio_reboot[multidut_port_info0-cold] 14:24:44 __init__.pytest_runtest_setup L0054 INFO | Before test: collected memory_values {'before_test': {}, 'after_test': {}} -------------------------- curtailed output ----------------------- 14:28:16 traffic_generation.run_traffic L0465 INFO | In-flight traffic statistics for flows: ['Test Flow Prio 3', 'Test Flow Prio 4', 'Background Flow Prio 1', 'Background Flow Prio 5', 'Background Flow Prio 6', 'Background Flow Prio 2', 'Background Flow Prio 0'] 14:28:16 traffic_generation.run_traffic L0466 INFO | In-flight TX frames: [68215, 68215, 116379310, 116379310, 116379310, 116379310, 116379310] 14:28:16 traffic_generation.run_traffic L0467 INFO | In-flight RX frames: [0, 0, 116379310, 116379310, 116379310, 116379310, 116379310] 14:28:49 traffic_generation.run_traffic L0468 INFO | DUT polling complete 14:28:49 traffic_generation.run_traffic L0479 INFO | Checking if all flows have stopped. Attempt sonic-net#1 14:28:50 traffic_generation.run_traffic L0486 INFO | All test and background traffic flows stopped 14:28:52 traffic_generation.run_traffic L0514 INFO | Dumping per-flow statistics 14:28:53 traffic_generation.run_traffic L0516 INFO | Stopping transmit on all remaining flows 14:29:00 snappi_api.info L1132 INFO | Flows stop 6.248s PASSED T2 Verification: AzDevOps@7c3002ea4994:/data/tests$ date;python3 -m pytest --inventory ../ansible/ixia-sonic --host-pattern board73,board74 --testbed ixre-chassis17-t2 --testbed_file ../ansible/testbed.csv --show-capture=stdout --log-cli-level info --showlocals -ra --allow_recover --junit-xml=/tmp/pfc/p.xml --skip_sanity --log-file=/tmp/pfc/p.log snappi_tests/pfc/test_pfc_pause_lossless_with_snappi.py -k reboot --pdb Wed Apr 30 20:44:41 UTC 2025 --------------- curtailed output ---------------- snappi_tests/pfc/test_pfc_pause_lossless_with_snappi.py::test_pfc_pause_single_lossless_prio_reboot[multidut_port_info0-cold-board71|3] --------------------------------------------------------------------------------------------------- live log setup ---------------------------------------------------------------------------------------------------- 20:45:40 __init__.set_default L0053 INFO | Completeness level not set during test execution. Setting to default level: CompletenessLevel.basic 20:45:40 __init__.check_test_completeness L0151 INFO | Test has no defined levels. Continue without test completeness checks 20:45:40 __init__.loganalyzer L0067 INFO | Log analyzer is disabled 20:45:40 __init__._fixture_generator_decorator L0081 INFO | -------------------- fixture ignore_route_check_for_cisco_8000 setup starts -------------------- 20:45:40 __init__._fixture_generator_decorator L0085 INFO | -------------------- fixture ignore_route_check_for_cisco_8000 setup ends -------------------- 20:45:40 __init__._fixture_generator_decorator L0081 INFO | -------------------- fixture multidut_port_info setup starts -------------------- 20:45:40 __init__._fixture_generator_decorator L0085 INFO | -------------------- fixture multidut_port_info setup ends -------------------- 20:45:40 __init__._fixture_generator_decorator L0081 INFO | -------------------- fixture number_of_tx_rx_ports setup starts -------------------- 20:45:40 __init__._fixture_generator_decorator L0085 INFO | -------------------- fixture number_of_tx_rx_ports setup ends -------------------- 20:45:40 __init__._fixture_generator_decorator L0081 INFO | -------------------- fixture setup_ports_and_dut setup starts -------------------- 20:45:40 helper.setup_ports_and_dut L0115 INFO | Running test for testbed subtype: multi-dut-long-to-short-link 20:45:43 snappi_fixtures.__intf_config_multidut L0858 INFO | Configuring Dut: board73 with port Ethernet16 with IP 20.1.1.2/31 20:45:43 snappi_fixtures.__intf_config_multidut L0878 INFO | Adding ip:20.1.1.2 on port:Ethernet16 on DUT:board73 20:45:44 snappi_fixtures.__intf_config_multidut L0858 INFO | Configuring Dut: board74 with port Ethernet0 with IP 20.1.1.0/31 20:45:44 snappi_fixtures.__intf_config_multidut L0878 INFO | Adding ip:20.1.1.0 on port:Ethernet0 on DUT:board74 20:45:45 __init__._fixture_generator_decorator L0085 INFO | -------------------- fixture setup_ports_and_dut setup ends -------------------- 20:45:45 __init__._fixture_generator_decorator L0081 INFO | -------------------- fixture reboot_duts_and_disable_wd setup starts -------------------- 20:45:46 sonic.get_asic_name L1778 INFO | asic: unknown 20:45:46 helper.skip_warm_reboot L0049 INFO | Reboot type cold is supported on broadcom switches 20:45:46 sonic.get_asic_name L1778 INFO | asic: unknown 20:45:46 helper.skip_warm_reboot L0049 INFO | Reboot type cold is supported on broadcom switches 20:45:49 helper.save_config_and_reboot L0408 INFO | Issuing a cold reboot on the dut board74 20:45:49 helper.save_config_and_reboot L0408 INFO | Issuing a cold reboot on the dut board73 20:45:52 reboot.reboot L0269 INFO | Reboot board74: wait[900], timeout[600] 20:45:52 reboot.reboot L0271 INFO | DUT board74 create a file /dev/shm/test_reboot before rebooting 20:45:52 reboot.reboot L0269 INFO | Reboot board73: wait[900], timeout[600] 20:45:52 reboot.reboot L0271 INFO | DUT board73 create a file /dev/shm/test_reboot before rebooting 20:45:54 reboot.wait_for_shutdown L0162 INFO | waiting for ssh to drop on board74 20:45:54 reboot.execute_reboot_command L0202 INFO | rebooting board74 with command "reboot" 20:45:54 reboot.wait_for_shutdown L0162 INFO | waiting for ssh to drop on board73 20:45:54 reboot.execute_reboot_command L0202 INFO | rebooting board73 with command "reboot" 20:46:15 reboot.wait_for_startup L0183 INFO | waiting for ssh to startup on board74 20:46:16 reboot.wait_for_startup L0183 INFO | waiting for ssh to startup on board73 20:48:38 reboot.wait_for_startup L0194 INFO | ssh has started up on board73 20:48:38 reboot.reboot L0288 INFO | waiting for switch board73 to initialize 20:48:40 reboot.wait_for_startup L0194 INFO | ssh has started up on board74 20:48:40 reboot.reboot L0288 INFO | waiting for switch board74 to initialize 20:52:24 processes_utils.wait_critical_processes L0079 INFO | Wait until all critical processes are healthy in 600 sec 20:52:24 processes_utils._all_critical_processes_ L0045 INFO | Check critical processes status 20:52:25 processes_utils.wait_critical_processes L0079 INFO | Wait until all critical processes are healthy in 600 sec 20:52:25 processes_utils._all_critical_processes_ L0045 INFO | Check critical processes status 20:52:32 processes_utils.get_critical_processes_s L0035 INFO | The status of checking process in container 'lldp0' is: False 20:52:32 processes_utils.get_critical_processes_s L0037 INFO | The processes not running in container 'lldp0' are: '['lldp-syncd', 'lldpmgrd']' 20:52:32 processes_utils.get_critical_processes_s L0035 INFO | The status of checking process in container 'lldp0' is: False 20:52:32 processes_utils.get_critical_processes_s L0037 INFO | The processes not running in container 'lldp0' are: '['lldp-syncd', 'lldpmgrd']' 20:52:52 processes_utils._all_critical_processes_ L0045 INFO | Check critical processes status 20:52:52 processes_utils._all_critical_processes_ L0045 INFO | Check critical processes status 20:52:59 processes_utils.get_critical_processes_s L0035 INFO | The status of checking process in container 'lldp0' is: False 20:52:59 processes_utils.get_critical_processes_s L0037 INFO | The processes not running in container 'lldp0' are: '['lldp-syncd', 'lldpmgrd']' 20:53:00 processes_utils.get_critical_processes_s L0035 INFO | The status of checking process in container 'lldp0' is: False 20:53:00 processes_utils.get_critical_processes_s L0037 INFO | The processes not running in container 'lldp0' are: '['lldp-syncd', 'lldpmgrd']' 20:53:20 processes_utils._all_critical_processes_ L0045 INFO | Check critical processes status 20:53:20 processes_utils._all_critical_processes_ L0045 INFO | Check critical processes status 20:53:27 reboot.reboot L0318 INFO | cold reboot finished on board73 20:53:28 reboot.reboot L0318 INFO | cold reboot finished on board74 20:53:28 reboot.reboot L0321 INFO | DUT board73 up since 2025-04-30 20:47:22.580000 20:53:28 reboot.reboot L0321 INFO | DUT board74 up since 2025-04-30 20:47:19.270000 20:53:29 helper.save_config_and_reboot L0412 INFO | Wait until the system is stable 20:53:29 helper.save_config_and_reboot L0412 INFO | Wait until the system is stable 20:53:35 sonic_asic.check_bgp_session_state L0692 INFO | bgp neighbors that match the state: ['3.3.3.1', '3.3.3.2', '3.3.3.6', '3.3.3.8', '3.3.3.12', '3.3.3.14', '3.3.3.20', '3333::3:1', '3333::3:2', '3333::3:6', '3333::3:8', '3333::3:12', '3333::3:14', '3333::3:20'] on namespace asic0 20:53:35 sonic_asic.check_bgp_session_state L0693 INFO | bgp neighbors to be checked on the state: [] on namespace asic0 20:53:35 sonic_asic.check_bgp_session_state L0692 INFO | bgp neighbors that match the state: ['3.3.3.1', '3.3.3.2', '3.3.3.6', '3.3.3.8', '3.3.3.14', '3.3.3.18', '3.3.3.20', '3333::3:1', '3333::3:2', '3333::3:6', '3333::3:8', '3333::3:14', '3333::3:18', '3333::3:20'] on namespace asic0 20:53:35 sonic_asic.check_bgp_session_state L0693 INFO | bgp neighbors to be checked on the state: [] on namespace asic0 20:53:36 sonic_asic.check_bgp_session_state L0692 INFO | bgp neighbors that match the state: ['3.3.3.1', '3.3.3.2', '3.3.3.6', '3.3.3.8', '3.3.3.12', '3.3.3.14', '3.3.3.18', '3333::3:1', '3333::3:2', '3333::3:6', '3333::3:8', '3333::3:12', '3333::3:14', '3333::3:18'] on namespace asic1 20:53:36 sonic_asic.check_bgp_session_state L0693 INFO | bgp neighbors to be checked on the state: [] on namespace asic1 20:53:36 parallel.on_terminate L0085 INFO | process save_config_and_reboot--<MultiAsicSonicHost board74> terminated with exit code None 20:53:36 sonic_asic.check_bgp_session_state L0692 INFO | bgp neighbors that match the state: ['3.3.3.1', '3.3.3.2', '3.3.3.6', '3.3.3.8', '3.3.3.12', '3.3.3.18', '3.3.3.20', '3333::3:1', '3333::3:2', '3333::3:6', '3333::3:8', '3333::3:12', '3333::3:18', '3333::3:20'] on namespace asic1 20:53:36 sonic_asic.check_bgp_session_state L0693 INFO | bgp neighbors to be checked on the state: [] on namespace asic1 20:53:36 parallel.on_terminate L0085 INFO | process save_config_and_reboot--<MultiAsicSonicHost board73> terminated with exit code None 20:53:36 parallel.parallel_run L0221 INFO | Completed running processes for target "save_config_and_reboot" in 0:07:49.938416 seconds 20:53:50 __init__._fixture_generator_decorator L0085 INFO | -------------------- fixture reboot_duts_and_disable_wd setup ends -------------------- --------------- curtailed output ---------------- PASSED [ 8%] -------------------------------------------------------------------------------------------------- live log teardown -------------------------------------------------------------------------------------------------- 20:57:53 __init__._fixture_generator_decorator L0093 INFO | -------------------- fixture reboot_duts_and_disable_wd teardown starts -------------------- 20:58:06 config_reload.config_reload L0145 INFO | reloading config_db 21:02:39 processes_utils.wait_critical_processes L0079 INFO | Wait until all critical processes are healthy in 600 sec 21:02:39 processes_utils._all_critical_processes_ L0045 INFO | Check critical processes status 21:02:46 processes_utils.get_critical_processes_s L0035 INFO | The status of checking process in container 'lldp0' is: False 21:02:46 processes_utils.get_critical_processes_s L0037 INFO | The processes not running in container 'lldp0' are: '['lldp-syncd', 'lldpmgrd']' 21:03:06 processes_utils._all_critical_processes_ L0045 INFO | Check critical processes status 21:03:14 config_reload.config_reload L0145 INFO | reloading config_db 21:07:44 processes_utils.wait_critical_processes L0079 INFO | Wait until all critical processes are healthy in 600 sec 21:07:44 processes_utils._all_critical_processes_ L0045 INFO | Check critical processes status 21:07:52 processes_utils.get_critical_processes_s L0035 INFO | The status of checking process in container 'lldp0' is: False 21:07:52 processes_utils.get_critical_processes_s L0037 INFO | The processes not running in container 'lldp0' are: '['lldp-syncd', 'lldpmgrd']' 21:08:12 processes_utils._all_critical_processes_ L0045 INFO | Check critical processes status 21:08:19 __init__._fixture_generator_decorator L0102 INFO | -------------------- fixture reboot_duts_and_disable_wd teardown ends -------------------- 21:08:19 __init__._fixture_generator_decorator L0093 INFO | -------------------- fixture setup_ports_and_dut teardown starts -------------------- 21:08:22 snappi_fixtures.__intf_config_multidut L0858 INFO | Configuring Dut: board73 with port Ethernet16 with IP 20.1.1.2/31 21:08:22 snappi_fixtures.__intf_config_multidut L0880 INFO | Removing ip:20.1.1.2 on port:Ethernet16 on DUT:board73 21:08:23 snappi_fixtures.__intf_config_multidut L0858 INFO | Configuring Dut: board74 with port Ethernet0 with IP 20.1.1.0/31 21:08:23 snappi_fixtures.__intf_config_multidut L0880 INFO | Removing ip:20.1.1.0 on port:Ethernet0 on DUT:board74 21:08:24 __init__._fixture_generator_decorator L0102 INFO | -------------------- fixture setup_ports_and_dut teardown ends -------------------- 21:08:24 __init__._fixture_generator_decorator L0093 INFO | -------------------- fixture number_of_tx_rx_ports teardown starts -------------------- 21:08:24 __init__._fixture_generator_decorator L0102 INFO | -------------------- fixture number_of_tx_rx_ports teardown ends -------------------- 21:08:24 __init__._fixture_generator_decorator L0093 INFO | -------------------- fixture multidut_port_info teardown starts -------------------- 21:08:24 __init__._fixture_generator_decorator L0102 INFO | -------------------- fixture multidut_port_info teardown ends -------------------- 21:08:24 __init__._fixture_generator_decorator L0093 INFO | -------------------- fixture ignore_route_check_for_cisco_8000 teardown starts -------------------- 21:08:24 __init__._fixture_generator_decorator L0102 INFO | -------------------- fixture ignore_route_check_for_cisco_8000 teardown ends -------------------- --------------- curtailed output ---------------- ======================================================================= 8 passed, 16 skipped, 20 deselected, 71 warnings in 10116.17s (2:48:36) ======================================================================= INFO:root:Can not get Allure report URL. Please check logs co-authorized by: jianquanye@microsoft.com Signed-off-by: Guy Shemesh <gshemesh@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description of PR
In tests/snappi_tests/pfc/test_pfc_pause_lossless_with_snappi.py, we have reboot testcases.
We disable the PFCWD and Credit-WD in the testcase and reboot the the DUT depending upon type of reboot. However, on cold-reboot, the Credit-WD is getting reset back.
This causes the lossless test flows on IXIA to continuously send traffic despite of the pause-storm. The run_traffic in traffic_generation.py is trying to determine if the test stops within a stipulated duration, which it does not and throws an assert that flows have not stopped within the given time duration and attempts.
Summary:
Fixes # (issue)
#18082
Type of change
Back port request
Approach
What is the motivation for this PR?
The reboot_dut was called after the PFCWD and Credit-WD disable. The reboot caused the Credit-WD to reset back and hence test failed.
How did you do it?
Created a single fixture called reboot_duts_and_disable_wd to reboot the DUTs and disable Credit and PFC watch-dogs.
This fixture is added in snappi_tests/files/helper.py.
Modified the testcase - snappi_tests/pfc/test_pfc_pause_lossless_with_snappi.py to incorporate the above fixture.
How did you verify/test it?
T0/T1 setup verification:
T2 Verification:
Any platform specific information?
Supported testbed topology if it's a new test case?
Documentation