[snappi]: Changed order of reboot and pfcwd_disable for reboot cases in pfc_pause_lossless test (sonic-net#18105)

Description of PR
In tests/snappi_tests/pfc/test_pfc_pause_lossless_with_snappi.py, we have reboot testcases.

We disable the PFCWD and Credit-WD in the testcase and reboot the the DUT depending upon type of reboot. However, on cold-reboot, the Credit-WD is getting reset back.

This causes the lossless test flows on IXIA to continuously send traffic despite of the pause-storm. The run_traffic in traffic_generation.py is trying to determine if the test stops within a stipulated duration, which it does not and throws an assert that flows have not stopped within the given time duration and attempts.

Summary:
Fixes # (issue)
sonic-net#18082

Type of change
 Bug fix
 Testbed and Framework(new/improvement)
 New Test case
 Skipped for non-supported platforms
 Test case improvement
Back port request
 202012
 202205
 202305
 202311
 202405
 202411
Approach
What is the motivation for this PR?
The reboot_dut was called after the PFCWD and Credit-WD disable. The reboot caused the Credit-WD to reset back and hence test failed.

How did you do it?
Created a single fixture called reboot_duts_and_disable_wd to reboot the DUTs and disable Credit and PFC watch-dogs.

This fixture is added in snappi_tests/files/helper.py.

Modified the testcase - snappi_tests/pfc/test_pfc_pause_lossless_with_snappi.py to incorporate the above fixture.

How did you verify/test it?
T0/T1 setup verification:

AzDevOps@sonic_mgmt_nokia_202411:/data/sonic-mgmt/tests$ date;python3 -m pytest --inventory ../ansible/ixia-sonic --host-pattern ixr-x3b-14 --testbed ixr-aaa-14-t2 --testbed_file ../ansible/testbed.csv --log-cli-level info --log-file-level info --kube_master unset --showlocals -ra --show-capture stdout --junit-xml=/tmp/p.xml --skip_sanity --log-file=/tmp/p.log --cache-clear --topology tgen snappi_tests/pfc/test_pfc_pause_lossless_with_snappi.py -k reboot
Thu May  1 13:45:24 UTC 2025

-------------------------- curtailed output -----------------------

snappi_tests/pfc/test_pfc_pause_lossless_with_snappi.py::test_pfc_pause_single_lossless_prio_reboot[multidut_port_info0-cold-ixr-aaa-14|3] 
--------------------------------------------------------------------------------------------------- live log setup ----------------------------------------------------------------------------------------------------
13:46:03 __init__.set_default                     L0053 INFO   | Completeness level not set during test execution. Setting to default level: CompletenessLevel.basic
13:46:03 __init__.check_test_completeness         L0151 INFO   | Test has no defined levels. Continue without test completeness checks
13:46:03 __init__.loganalyzer                     L0067 INFO   | Log analyzer is disabled
13:46:03 __init__.store_fixture_values            L0020 INFO   | store memory_utilization test_pfc_pause_single_lossless_prio_reboot[multidut_port_info0-cold-ixr-aaa-14|3]
13:46:03 __init__._fixture_generator_decorator    L0081 INFO   | -------------------- fixture ignore_route_check_for_cisco_8000 setup starts --------------------
13:46:03 __init__._fixture_generator_decorator    L0085 INFO   | -------------------- fixture ignore_route_check_for_cisco_8000 setup ends --------------------
13:46:03 __init__._fixture_generator_decorator    L0081 INFO   | -------------------- fixture multidut_port_info setup starts --------------------
13:46:03 __init__._fixture_generator_decorator    L0085 INFO   | -------------------- fixture multidut_port_info setup ends --------------------
13:46:03 __init__._fixture_generator_decorator    L0081 INFO   | -------------------- fixture number_of_tx_rx_ports setup starts --------------------
13:46:03 __init__._fixture_generator_decorator    L0085 INFO   | -------------------- fixture number_of_tx_rx_ports setup ends --------------------
13:46:03 __init__._fixture_generator_decorator    L0081 INFO   | -------------------- fixture setup_ports_and_dut setup starts --------------------
13:46:03 helper.setup_ports_and_dut               L0111 INFO   | Running test for testbed subtype: single-dut-multi-asic-aaa
13:46:04 __init__._fixture_generator_decorator    L0085 INFO   | -------------------- fixture setup_ports_and_dut setup ends --------------------
13:46:04 __init__._fixture_generator_decorator    L0081 INFO   | -------------------- fixture reboot_duts_and_disable_wd setup starts --------------------
13:46:05 sonic.get_asic_name                      L1797 INFO   | asic: unknown
13:46:05 helper.skip_warm_reboot                  L0045 INFO   | Reboot type cold is  supported on broadcom switches
13:46:05 sonic.get_asic_name                      L1797 INFO   | asic: unknown
13:46:05 helper.skip_warm_reboot                  L0045 INFO   | Reboot type cold is  supported on broadcom switches
13:46:08 helper.save_config_and_reboot            L0210 INFO   | Issuing a cold reboot on the dut ixr-aaa-14
13:46:11 reboot.reboot                            L0271 INFO   | Reboot ixr-aaa-14: wait[120], timeout[300]
13:46:11 reboot.reboot                            L0273 INFO   | DUT ixr-aaa-14 create a file /dev/shm/test_reboot before rebooting
13:46:12 reboot.collect_console_log               L0566 INFO   | start: collect console log
13:46:12 dut_utils.creds_on_dut                   L0471 INFO   | dut ixr-aaa-14 belongs to groups ['ixia-sonic', 'sonic', 'sonic_aaa', 'fanout']
13:46:12 dut_utils.creds_on_dut                   L0496 INFO   | skip empty var file /data/sonic-mgmt/tests/common/helpers/../../../ansible/group_vars/all/corefile_uploader.yml
13:46:12 transport._log                           L1873 INFO   | Connected (version 2.0, client OpenSSH_9.2p1)
13:46:16 transport._log                           L1873 INFO   | Auth banner: b'Debian GNU/Linux 12 \\n \\l\n\n'
13:46:16 transport._log                           L1873 INFO   | Authentication (password) failed.
13:46:16 transport._log                           L1873 INFO   | Connected (version 2.0, client OpenSSH_9.2p1)
13:46:16 transport._log                           L1873 INFO   | Auth banner: b'Debian GNU/Linux 12 \\n \\l\n\n'
13:46:16 transport._log                           L1873 INFO   | Authentication (password) successful!
13:46:17 reboot.try_create_dut_console            L0559 WARNING| Fail to create dut console. Please check console config or if console works ro not. 'ManagementIp'
13:46:17 reboot.collect_console_log               L0576 WARNING| dut console is not ready, we cannot get log by console
13:46:17 reboot.wait_for_shutdown                 L0164 INFO   | waiting for ssh to drop on ixr-aaa-14
13:46:17 reboot.execute_reboot_command            L0205 INFO   | rebooting ixr-aaa-14 with command "reboot"
13:46:35 reboot.wait_for_startup                  L0185 INFO   | waiting for ssh to startup on ixr-aaa-14
13:48:05 reboot.wait_for_startup                  L0197 INFO   | ssh has started up on ixr-aaa-14
13:48:05 reboot.reboot                            L0300 INFO   | waiting for switch ixr-aaa-14 to initialize
13:52:31 processes_utils.wait_critical_processes  L0078 INFO   | Wait until all critical processes are healthy in 600 sec
13:52:31 processes_utils._all_critical_processes_ L0045 INFO   | Check critical processes status
13:52:40 reboot.reboot                            L0330 INFO   | cold reboot finished on ixr-aaa-14
13:52:41 reboot.reboot                            L0333 INFO   | DUT ixr-aaa-14 up since 2025-05-01 13:47:31.410000
13:52:41 helper.save_config_and_reboot            L0214 INFO   | Wait until the system is stable
13:52:47 sonic_asic.check_bgp_session_state       L0714 INFO   | bgp neighbors that match the state: ['3.3.3.2', '3333::3:2'] on namespace asic0
13:52:47 sonic_asic.check_bgp_session_state       L0715 INFO   | bgp neighbors to be checked on the state: [] on namespace asic0
13:52:48 sonic_asic.check_bgp_session_state       L0714 INFO   | bgp neighbors that match the state: ['3.3.3.1', '3333::3:1'] on namespace asic1
13:52:48 sonic_asic.check_bgp_session_state       L0715 INFO   | bgp neighbors to be checked on the state: [] on namespace asic1
13:52:48 parallel.on_terminate                    L0085 INFO   | process save_config_and_reboot--<MultiAsicSonicHost ixr-aaa-14> terminated with exit code None
13:52:48 parallel.parallel_run                    L0221 INFO   | Completed running processes for target "save_config_and_reboot" in 0:06:43.000484 seconds
13:52:54 __init__._fixture_generator_decorator    L0085 INFO   | -------------------- fixture reboot_duts_and_disable_wd setup ends --------------------
13:52:54 __init__.pytest_runtest_setup            L0034 INFO   | collect memory before test test_pfc_pause_single_lossless_prio_reboot[multidut_port_info0-cold-ixr-aaa-14|3]
13:52:54 __init__.pytest_runtest_setup            L0054 INFO   | Before test: collected memory_values {'before_test': {}, 'after_test': {}}

-------------------------- curtailed output -----------------------

13:57:03 traffic_generation.run_traffic           L0465 INFO   | In-flight traffic statistics for flows: ['Test Flow Prio 3', 'Background Flow Prio 1', 'Background Flow Prio 4', 'Background Flow Prio 5', 'Background Flow Prio 6', 'Background Flow Prio 2', 'Background Flow Prio 0']
13:57:03 traffic_generation.run_traffic           L0466 INFO   | In-flight TX frames: [70428, 63697318, 63697318, 63697318, 63697318, 63697318, 63697318]
13:57:03 traffic_generation.run_traffic           L0467 INFO   | In-flight RX frames: [0, 63697318, 63697318, 63697318, 63697318, 63697318, 63697318]
13:57:21 traffic_generation.run_traffic           L0468 INFO   | DUT polling complete
13:57:21 traffic_generation.run_traffic           L0479 INFO   | Checking if all flows have stopped. Attempt #1
13:57:22 traffic_generation.run_traffic           L0486 INFO   | All test and background traffic flows stopped
13:57:24 traffic_generation.run_traffic           L0514 INFO   | Dumping per-flow statistics
13:57:25 traffic_generation.run_traffic           L0516 INFO   | Stopping transmit on all remaining flows
13:57:32 snappi_api.info                          L1132 INFO   | Flows stop 6.258s
PASSED        

-------------------------- curtailed output -----------------------

snappi_tests/pfc/test_pfc_pause_lossless_with_snappi.py::test_pfc_pause_multi_lossless_prio_reboot[multidut_port_info0-cold] 
--------------------------------------------------------------------------------------------------- live log setup ----------------------------------------------------------------------------------------------------
14:17:56 __init__.set_default                     L0053 INFO   | Completeness level not set during test execution. Setting to default level: CompletenessLevel.basic
14:17:56 __init__.check_test_completeness         L0151 INFO   | Test has no defined levels. Continue without test completeness checks
14:17:56 __init__.loganalyzer                     L0067 INFO   | Log analyzer is disabled
14:17:56 __init__.store_fixture_values            L0020 INFO   | store memory_utilization test_pfc_pause_multi_lossless_prio_reboot[multidut_port_info0-cold]
14:17:56 __init__._fixture_generator_decorator    L0081 INFO   | -------------------- fixture ignore_route_check_for_cisco_8000 setup starts --------------------
14:17:56 __init__._fixture_generator_decorator    L0085 INFO   | -------------------- fixture ignore_route_check_for_cisco_8000 setup ends --------------------
14:17:56 __init__._fixture_generator_decorator    L0081 INFO   | -------------------- fixture multidut_port_info setup starts --------------------
14:17:56 __init__._fixture_generator_decorator    L0085 INFO   | -------------------- fixture multidut_port_info setup ends --------------------
14:17:56 __init__._fixture_generator_decorator    L0081 INFO   | -------------------- fixture number_of_tx_rx_ports setup starts --------------------
14:17:56 __init__._fixture_generator_decorator    L0085 INFO   | -------------------- fixture number_of_tx_rx_ports setup ends --------------------
14:17:56 __init__._fixture_generator_decorator    L0081 INFO   | -------------------- fixture setup_ports_and_dut setup starts --------------------
14:17:56 helper.setup_ports_and_dut               L0111 INFO   | Running test for testbed subtype: single-dut-multi-asic-aaa
14:17:57 __init__._fixture_generator_decorator    L0085 INFO   | -------------------- fixture setup_ports_and_dut setup ends --------------------
14:17:57 __init__._fixture_generator_decorator    L0081 INFO   | -------------------- fixture reboot_duts_and_disable_wd setup starts --------------------
14:17:58 sonic.get_asic_name                      L1797 INFO   | asic: unknown
14:17:58 helper.skip_warm_reboot                  L0045 INFO   | Reboot type cold is  supported on broadcom switches
14:17:58 sonic.get_asic_name                      L1797 INFO   | asic: unknown
14:17:58 helper.skip_warm_reboot                  L0045 INFO   | Reboot type cold is  supported on broadcom switches
14:18:01 helper.save_config_and_reboot            L0210 INFO   | Issuing a cold reboot on the dut ixr-aaa-14
14:18:04 reboot.reboot                            L0271 INFO   | Reboot ixr-aaa-14: wait[120], timeout[300]
14:18:04 reboot.reboot                            L0273 INFO   | DUT ixr-aaa-14 create a file /dev/shm/test_reboot before rebooting
14:18:05 reboot.collect_console_log               L0566 INFO   | start: collect console log
14:18:05 dut_utils.creds_on_dut                   L0471 INFO   | dut ixr-aaa-14 belongs to groups ['ixia-sonic', 'sonic', 'sonic_aaa', 'fanout']
14:18:05 dut_utils.creds_on_dut                   L0496 INFO   | skip empty var file /data/sonic-mgmt/tests/common/helpers/../../../ansible/group_vars/all/corefile_uploader.yml
14:18:05 transport._log                           L1873 INFO   | Connected (version 2.0, client OpenSSH_9.2p1)
14:18:07 transport._log                           L1873 INFO   | Auth banner: b'Debian GNU/Linux 12 \\n \\l\n\n'
14:18:07 transport._log                           L1873 INFO   | Authentication (password) failed.
14:18:07 transport._log                           L1873 INFO   | Connected (version 2.0, client OpenSSH_9.2p1)
14:18:07 transport._log                           L1873 INFO   | Auth banner: b'Debian GNU/Linux 12 \\n \\l\n\n'
14:18:07 transport._log                           L1873 INFO   | Authentication (password) successful!
14:18:08 reboot.try_create_dut_console            L0559 WARNING| Fail to create dut console. Please check console config or if console works ro not. 'ManagementIp'
14:18:08 reboot.collect_console_log               L0576 WARNING| dut console is not ready, we cannot get log by console
14:18:10 reboot.wait_for_shutdown                 L0164 INFO   | waiting for ssh to drop on ixr-aaa-14
14:18:10 reboot.execute_reboot_command            L0205 INFO   | rebooting ixr-aaa-14 with command "reboot"
14:18:28 reboot.wait_for_startup                  L0185 INFO   | waiting for ssh to startup on ixr-aaa-14
14:19:56 reboot.wait_for_startup                  L0197 INFO   | ssh has started up on ixr-aaa-14
14:19:56 reboot.reboot                            L0300 INFO   | waiting for switch ixr-aaa-14 to initialize
14:24:22 processes_utils.wait_critical_processes  L0078 INFO   | Wait until all critical processes are healthy in 600 sec
14:24:22 processes_utils._all_critical_processes_ L0045 INFO   | Check critical processes status
14:24:30 reboot.reboot                            L0330 INFO   | cold reboot finished on ixr-aaa-14
14:24:31 reboot.reboot                            L0333 INFO   | DUT ixr-aaa-14 up since 2025-05-01 14:19:23.320000
14:24:31 helper.save_config_and_reboot            L0214 INFO   | Wait until the system is stable
14:24:37 sonic_asic.check_bgp_session_state       L0714 INFO   | bgp neighbors that match the state: ['3.3.3.2', '3333::3:2'] on namespace asic0
14:24:37 sonic_asic.check_bgp_session_state       L0715 INFO   | bgp neighbors to be checked on the state: [] on namespace asic0
14:24:38 sonic_asic.check_bgp_session_state       L0714 INFO   | bgp neighbors that match the state: ['3.3.3.1', '3333::3:1'] on namespace asic1
14:24:38 sonic_asic.check_bgp_session_state       L0715 INFO   | bgp neighbors to be checked on the state: [] on namespace asic1
14:24:38 parallel.on_terminate                    L0085 INFO   | process save_config_and_reboot--<MultiAsicSonicHost ixr-aaa-14> terminated with exit code None
14:24:38 parallel.parallel_run                    L0221 INFO   | Completed running processes for target "save_config_and_reboot" in 0:06:39.645613 seconds
14:24:44 __init__._fixture_generator_decorator    L0085 INFO   | -------------------- fixture reboot_duts_and_disable_wd setup ends --------------------
14:24:44 __init__.pytest_runtest_setup            L0034 INFO   | collect memory before test test_pfc_pause_multi_lossless_prio_reboot[multidut_port_info0-cold]
14:24:44 __init__.pytest_runtest_setup            L0054 INFO   | Before test: collected memory_values {'before_test': {}, 'after_test': {}}
-------------------------- curtailed output -----------------------
14:28:16 traffic_generation.run_traffic           L0465 INFO   | In-flight traffic statistics for flows: ['Test Flow Prio 3', 'Test Flow Prio 4', 'Background Flow Prio 1', 'Background Flow Prio 5', 'Background Flow Prio 6', 'Background Flow Prio 2', 'Background Flow Prio 0']
14:28:16 traffic_generation.run_traffic           L0466 INFO   | In-flight TX frames: [68215, 68215, 116379310, 116379310, 116379310, 116379310, 116379310]
14:28:16 traffic_generation.run_traffic           L0467 INFO   | In-flight RX frames: [0, 0, 116379310, 116379310, 116379310, 116379310, 116379310]
14:28:49 traffic_generation.run_traffic           L0468 INFO   | DUT polling complete
14:28:49 traffic_generation.run_traffic           L0479 INFO   | Checking if all flows have stopped. Attempt #1
14:28:50 traffic_generation.run_traffic           L0486 INFO   | All test and background traffic flows stopped
14:28:52 traffic_generation.run_traffic           L0514 INFO   | Dumping per-flow statistics
14:28:53 traffic_generation.run_traffic           L0516 INFO   | Stopping transmit on all remaining flows
14:29:00 snappi_api.info                          L1132 INFO   | Flows stop 6.248s
PASSED           
T2 Verification:

AzDevOps@7c3002ea4994:/data/tests$ date;python3 -m pytest --inventory ../ansible/ixia-sonic --host-pattern board73,board74 --testbed ixre-chassis17-t2 --testbed_file ../ansible/testbed.csv --show-capture=stdout --log-cli-level info --showlocals -ra --allow_recover --junit-xml=/tmp/pfc/p.xml --skip_sanity --log-file=/tmp/pfc/p.log snappi_tests/pfc/test_pfc_pause_lossless_with_snappi.py -k reboot --pdb        
Wed Apr 30 20:44:41 UTC 2025

--------------- curtailed output ----------------

snappi_tests/pfc/test_pfc_pause_lossless_with_snappi.py::test_pfc_pause_single_lossless_prio_reboot[multidut_port_info0-cold-board71|3] 
--------------------------------------------------------------------------------------------------- live log setup ----------------------------------------------------------------------------------------------------
20:45:40 __init__.set_default                     L0053 INFO   | Completeness level not set during test execution. Setting to default level: CompletenessLevel.basic
20:45:40 __init__.check_test_completeness         L0151 INFO   | Test has no defined levels. Continue without test completeness checks
20:45:40 __init__.loganalyzer                     L0067 INFO   | Log analyzer is disabled
20:45:40 __init__._fixture_generator_decorator    L0081 INFO   | -------------------- fixture ignore_route_check_for_cisco_8000 setup starts --------------------
20:45:40 __init__._fixture_generator_decorator    L0085 INFO   | -------------------- fixture ignore_route_check_for_cisco_8000 setup ends --------------------
20:45:40 __init__._fixture_generator_decorator    L0081 INFO   | -------------------- fixture multidut_port_info setup starts --------------------
20:45:40 __init__._fixture_generator_decorator    L0085 INFO   | -------------------- fixture multidut_port_info setup ends --------------------
20:45:40 __init__._fixture_generator_decorator    L0081 INFO   | -------------------- fixture number_of_tx_rx_ports setup starts --------------------
20:45:40 __init__._fixture_generator_decorator    L0085 INFO   | -------------------- fixture number_of_tx_rx_ports setup ends --------------------
20:45:40 __init__._fixture_generator_decorator    L0081 INFO   | -------------------- fixture setup_ports_and_dut setup starts --------------------
20:45:40 helper.setup_ports_and_dut               L0115 INFO   | Running test for testbed subtype: multi-dut-long-to-short-link
20:45:43 snappi_fixtures.__intf_config_multidut   L0858 INFO   | Configuring Dut: board73 with port Ethernet16 with IP 20.1.1.2/31
20:45:43 snappi_fixtures.__intf_config_multidut   L0878 INFO   | Adding ip:20.1.1.2 on port:Ethernet16 on DUT:board73
20:45:44 snappi_fixtures.__intf_config_multidut   L0858 INFO   | Configuring Dut: board74 with port Ethernet0 with IP 20.1.1.0/31
20:45:44 snappi_fixtures.__intf_config_multidut   L0878 INFO   | Adding ip:20.1.1.0 on port:Ethernet0 on DUT:board74
20:45:45 __init__._fixture_generator_decorator    L0085 INFO   | -------------------- fixture setup_ports_and_dut setup ends --------------------
20:45:45 __init__._fixture_generator_decorator    L0081 INFO   | -------------------- fixture reboot_duts_and_disable_wd setup starts --------------------
20:45:46 sonic.get_asic_name                      L1778 INFO   | asic: unknown
20:45:46 helper.skip_warm_reboot                  L0049 INFO   | Reboot type cold is  supported on broadcom switches
20:45:46 sonic.get_asic_name                      L1778 INFO   | asic: unknown
20:45:46 helper.skip_warm_reboot                  L0049 INFO   | Reboot type cold is  supported on broadcom switches
20:45:49 helper.save_config_and_reboot            L0408 INFO   | Issuing a cold reboot on the dut board74
20:45:49 helper.save_config_and_reboot            L0408 INFO   | Issuing a cold reboot on the dut board73
20:45:52 reboot.reboot                            L0269 INFO   | Reboot board74: wait[900], timeout[600]
20:45:52 reboot.reboot                            L0271 INFO   | DUT board74 create a file /dev/shm/test_reboot before rebooting
20:45:52 reboot.reboot                            L0269 INFO   | Reboot board73: wait[900], timeout[600]
20:45:52 reboot.reboot                            L0271 INFO   | DUT board73 create a file /dev/shm/test_reboot before rebooting
20:45:54 reboot.wait_for_shutdown                 L0162 INFO   | waiting for ssh to drop on board74
20:45:54 reboot.execute_reboot_command            L0202 INFO   | rebooting board74 with command "reboot"
20:45:54 reboot.wait_for_shutdown                 L0162 INFO   | waiting for ssh to drop on board73
20:45:54 reboot.execute_reboot_command            L0202 INFO   | rebooting board73 with command "reboot"
20:46:15 reboot.wait_for_startup                  L0183 INFO   | waiting for ssh to startup on board74
20:46:16 reboot.wait_for_startup                  L0183 INFO   | waiting for ssh to startup on board73
20:48:38 reboot.wait_for_startup                  L0194 INFO   | ssh has started up on board73
20:48:38 reboot.reboot                            L0288 INFO   | waiting for switch board73 to initialize
20:48:40 reboot.wait_for_startup                  L0194 INFO   | ssh has started up on board74
20:48:40 reboot.reboot                            L0288 INFO   | waiting for switch board74 to initialize
20:52:24 processes_utils.wait_critical_processes  L0079 INFO   | Wait until all critical processes are healthy in 600 sec
20:52:24 processes_utils._all_critical_processes_ L0045 INFO   | Check critical processes status
20:52:25 processes_utils.wait_critical_processes  L0079 INFO   | Wait until all critical processes are healthy in 600 sec
20:52:25 processes_utils._all_critical_processes_ L0045 INFO   | Check critical processes status
20:52:32 processes_utils.get_critical_processes_s L0035 INFO   | The status of checking process in container 'lldp0' is: False
20:52:32 processes_utils.get_critical_processes_s L0037 INFO   | The processes not running in container 'lldp0' are: '['lldp-syncd', 'lldpmgrd']'
20:52:32 processes_utils.get_critical_processes_s L0035 INFO   | The status of checking process in container 'lldp0' is: False
20:52:32 processes_utils.get_critical_processes_s L0037 INFO   | The processes not running in container 'lldp0' are: '['lldp-syncd', 'lldpmgrd']'
20:52:52 processes_utils._all_critical_processes_ L0045 INFO   | Check critical processes status
20:52:52 processes_utils._all_critical_processes_ L0045 INFO   | Check critical processes status
20:52:59 processes_utils.get_critical_processes_s L0035 INFO   | The status of checking process in container 'lldp0' is: False
20:52:59 processes_utils.get_critical_processes_s L0037 INFO   | The processes not running in container 'lldp0' are: '['lldp-syncd', 'lldpmgrd']'
20:53:00 processes_utils.get_critical_processes_s L0035 INFO   | The status of checking process in container 'lldp0' is: False
20:53:00 processes_utils.get_critical_processes_s L0037 INFO   | The processes not running in container 'lldp0' are: '['lldp-syncd', 'lldpmgrd']'
20:53:20 processes_utils._all_critical_processes_ L0045 INFO   | Check critical processes status
20:53:20 processes_utils._all_critical_processes_ L0045 INFO   | Check critical processes status
20:53:27 reboot.reboot                            L0318 INFO   | cold reboot finished on board73
20:53:28 reboot.reboot                            L0318 INFO   | cold reboot finished on board74
20:53:28 reboot.reboot                            L0321 INFO   | DUT board73 up since 2025-04-30 20:47:22.580000
20:53:28 reboot.reboot                            L0321 INFO   | DUT board74 up since 2025-04-30 20:47:19.270000
20:53:29 helper.save_config_and_reboot            L0412 INFO   | Wait until the system is stable
20:53:29 helper.save_config_and_reboot            L0412 INFO   | Wait until the system is stable
20:53:35 sonic_asic.check_bgp_session_state       L0692 INFO   | bgp neighbors that match the state: ['3.3.3.1', '3.3.3.2', '3.3.3.6', '3.3.3.8', '3.3.3.12', '3.3.3.14', '3.3.3.20', '3333::3:1', '3333::3:2', '3333::3:6', '3333::3:8', '3333::3:12', '3333::3:14', '3333::3:20'] on namespace asic0
20:53:35 sonic_asic.check_bgp_session_state       L0693 INFO   | bgp neighbors to be checked on the state: [] on namespace asic0
20:53:35 sonic_asic.check_bgp_session_state       L0692 INFO   | bgp neighbors that match the state: ['3.3.3.1', '3.3.3.2', '3.3.3.6', '3.3.3.8', '3.3.3.14', '3.3.3.18', '3.3.3.20', '3333::3:1', '3333::3:2', '3333::3:6', '3333::3:8', '3333::3:14', '3333::3:18', '3333::3:20'] on namespace asic0
20:53:35 sonic_asic.check_bgp_session_state       L0693 INFO   | bgp neighbors to be checked on the state: [] on namespace asic0
20:53:36 sonic_asic.check_bgp_session_state       L0692 INFO   | bgp neighbors that match the state: ['3.3.3.1', '3.3.3.2', '3.3.3.6', '3.3.3.8', '3.3.3.12', '3.3.3.14', '3.3.3.18', '3333::3:1', '3333::3:2', '3333::3:6', '3333::3:8', '3333::3:12', '3333::3:14', '3333::3:18'] on namespace asic1
20:53:36 sonic_asic.check_bgp_session_state       L0693 INFO   | bgp neighbors to be checked on the state: [] on namespace asic1
20:53:36 parallel.on_terminate                    L0085 INFO   | process save_config_and_reboot--<MultiAsicSonicHost board74> terminated with exit code None
20:53:36 sonic_asic.check_bgp_session_state       L0692 INFO   | bgp neighbors that match the state: ['3.3.3.1', '3.3.3.2', '3.3.3.6', '3.3.3.8', '3.3.3.12', '3.3.3.18', '3.3.3.20', '3333::3:1', '3333::3:2', '3333::3:6', '3333::3:8', '3333::3:12', '3333::3:18', '3333::3:20'] on namespace asic1
20:53:36 sonic_asic.check_bgp_session_state       L0693 INFO   | bgp neighbors to be checked on the state: [] on namespace asic1
20:53:36 parallel.on_terminate                    L0085 INFO   | process save_config_and_reboot--<MultiAsicSonicHost board73> terminated with exit code None
20:53:36 parallel.parallel_run                    L0221 INFO   | Completed running processes for target "save_config_and_reboot" in 0:07:49.938416 seconds
20:53:50 __init__._fixture_generator_decorator    L0085 INFO   | -------------------- fixture reboot_duts_and_disable_wd setup ends --------------------

--------------- curtailed output ----------------

PASSED                                                                                                                                                                                                          [  8%]
-------------------------------------------------------------------------------------------------- live log teardown --------------------------------------------------------------------------------------------------
20:57:53 __init__._fixture_generator_decorator    L0093 INFO   | -------------------- fixture reboot_duts_and_disable_wd teardown starts --------------------
20:58:06 config_reload.config_reload              L0145 INFO   | reloading config_db
21:02:39 processes_utils.wait_critical_processes  L0079 INFO   | Wait until all critical processes are healthy in 600 sec
21:02:39 processes_utils._all_critical_processes_ L0045 INFO   | Check critical processes status
21:02:46 processes_utils.get_critical_processes_s L0035 INFO   | The status of checking process in container 'lldp0' is: False
21:02:46 processes_utils.get_critical_processes_s L0037 INFO   | The processes not running in container 'lldp0' are: '['lldp-syncd', 'lldpmgrd']'
21:03:06 processes_utils._all_critical_processes_ L0045 INFO   | Check critical processes status
21:03:14 config_reload.config_reload              L0145 INFO   | reloading config_db
21:07:44 processes_utils.wait_critical_processes  L0079 INFO   | Wait until all critical processes are healthy in 600 sec
21:07:44 processes_utils._all_critical_processes_ L0045 INFO   | Check critical processes status
21:07:52 processes_utils.get_critical_processes_s L0035 INFO   | The status of checking process in container 'lldp0' is: False
21:07:52 processes_utils.get_critical_processes_s L0037 INFO   | The processes not running in container 'lldp0' are: '['lldp-syncd', 'lldpmgrd']'
21:08:12 processes_utils._all_critical_processes_ L0045 INFO   | Check critical processes status
21:08:19 __init__._fixture_generator_decorator    L0102 INFO   | -------------------- fixture reboot_duts_and_disable_wd teardown ends --------------------
21:08:19 __init__._fixture_generator_decorator    L0093 INFO   | -------------------- fixture setup_ports_and_dut teardown starts --------------------
21:08:22 snappi_fixtures.__intf_config_multidut   L0858 INFO   | Configuring Dut: board73 with port Ethernet16 with IP 20.1.1.2/31
21:08:22 snappi_fixtures.__intf_config_multidut   L0880 INFO   | Removing ip:20.1.1.2 on port:Ethernet16 on DUT:board73
21:08:23 snappi_fixtures.__intf_config_multidut   L0858 INFO   | Configuring Dut: board74 with port Ethernet0 with IP 20.1.1.0/31
21:08:23 snappi_fixtures.__intf_config_multidut   L0880 INFO   | Removing ip:20.1.1.0 on port:Ethernet0 on DUT:board74
21:08:24 __init__._fixture_generator_decorator    L0102 INFO   | -------------------- fixture setup_ports_and_dut teardown ends --------------------
21:08:24 __init__._fixture_generator_decorator    L0093 INFO   | -------------------- fixture number_of_tx_rx_ports teardown starts --------------------
21:08:24 __init__._fixture_generator_decorator    L0102 INFO   | -------------------- fixture number_of_tx_rx_ports teardown ends --------------------
21:08:24 __init__._fixture_generator_decorator    L0093 INFO   | -------------------- fixture multidut_port_info teardown starts --------------------
21:08:24 __init__._fixture_generator_decorator    L0102 INFO   | -------------------- fixture multidut_port_info teardown ends --------------------
21:08:24 __init__._fixture_generator_decorator    L0093 INFO   | -------------------- fixture ignore_route_check_for_cisco_8000 teardown starts --------------------
21:08:24 __init__._fixture_generator_decorator    L0102 INFO   | -------------------- fixture ignore_route_check_for_cisco_8000 teardown ends --------------------

--------------- curtailed output ----------------

======================================================================= 8 passed, 16 skipped, 20 deselected, 71 warnings in 10116.17s (2:48:36) =======================================================================
INFO:root:Can not get Allure report URL. Please check logs

co-authorized by: [email protected]

Author: amitpawar12

tests/snappi_tests/files/helper.py

@@ -14,6 +14,10 @@
 from tests.common.platform.interface_utils import check_interface_status_of_up_ports
 from tests.common.snappi_tests.snappi_fixtures import get_snappi_ports_for_rdma, \
     snappi_dut_base_config, is_snappi_multidut
+from tests.common.snappi_tests.qos_fixtures import reapply_pfcwd, get_pfcwd_config
+from tests.common.snappi_tests.common_helpers import \
+        stop_pfcwd, disable_packet_aging, enable_packet_aging
+
 
 logger = logging.getLogger(__name__)
 
@@ -383,3 +387,50 @@ def get_npu_voq_queue_counters(duthost, interface, priority, clear=False):
         dict_output[entry] = value
 
     return dict_output
+
+
+@pytest.fixture(params=['warm', 'cold', 'fast'])
+def reboot_duts_and_disable_wd(setup_ports_and_dut, localhost, request):
+    '''
+    Purpose of the function is to have reboot_duts and disable watchdogs.
+    '''
+    reboot_type = request.param
+    _, _, snappi_ports = setup_ports_and_dut
+    skip_warm_reboot(snappi_ports[0]['duthost'], reboot_type)
+    skip_warm_reboot(snappi_ports[1]['duthost'], reboot_type)
+
+    def save_config_and_reboot(node, results=None):
+        up_bgp_neighbors = node.get_bgp_neighbors_per_asic("established")
+        logger.info("Issuing a {} reboot on the dut {}".format(reboot_type, node.hostname))
+        node.shell("mkdir /etc/sonic/orig_configs; mv /etc/sonic/config_db* /etc/sonic/orig_configs/")
+        node.shell("sudo config save -y")
+        reboot(node, localhost, reboot_type=reboot_type, safe_reboot=True)
+        logger.info("Wait until the system is stable")
+        wait_until(180, 20, 0, node.critical_services_fully_started)
+        wait_until(180, 20, 0, check_interface_status_of_up_ports, node)
+        wait_until(300, 10, 0, node.check_bgp_session_state_all_asics, up_bgp_neighbors, "established")
+
+    # Convert the list of duthosts into a list of tuples as required for parallel func.
+    args = set((snappi_ports[0]['duthost'], snappi_ports[1]['duthost']))
+    parallel_run(save_config_and_reboot, {}, {}, list(args), timeout=900)
+
+    pfcwd_value = {}
+
+    for duthost in list(args):
+        pfcwd_value[duthost.hostname] = get_pfcwd_config(duthost)
+        stop_pfcwd(duthost)
+        disable_packet_aging(duthost)
+
+    yield
+
+    for duthost in list(args):
+        reapply_pfcwd(duthost, pfcwd_value[duthost.hostname])
+        enable_packet_aging(duthost)
+
+    def revert_config_and_reload(node, results=None):
+        node.shell("mv /etc/sonic/orig_configs/* /etc/sonic/ ; rmdir /etc/sonic/orig_configs; ")
+        config_reload(node, safe_reload=True)
+
+    # parallel_run(revert_config_and_reload, {}, {}, list(args), timeout=900)
+    for duthost in args:
+        revert_config_and_reload(node=duthost)

tests/snappi_tests/pfc/test_pfc_pause_lossless_with_snappi.py

@@ -6,7 +6,8 @@
     snappi_api, snappi_dut_base_config, get_snappi_ports_for_rdma, cleanup_config, get_snappi_ports_multi_dut, \
     snappi_testbed_config, get_snappi_ports_single_dut, \
     get_snappi_ports, is_snappi_multidut                                        # noqa: F401
-from tests.snappi_tests.files.helper import multidut_port_info, setup_ports_and_dut, reboot_duts  # noqa: F401
+from tests.snappi_tests.files.helper import multidut_port_info, setup_ports_and_dut, reboot_duts, \
+    reboot_duts_and_disable_wd                                              # noqa: F401
 from tests.common.snappi_tests.qos_fixtures import prio_dscp_map, all_prio_list, lossless_prio_list,\
     lossy_prio_list, disable_pfcwd          # noqa F401
 from tests.snappi_tests.pfc.files.helper import run_pfc_test
@@ -204,14 +205,13 @@ def test_pfc_pause_single_lossless_prio_reboot(snappi_api,                   # n
                                                duthosts,
                                                localhost,
                                                enum_one_dut_lossless_prio_with_completeness_level,    # noqa: F811
-                                               prio_dscp_map,            # noqa: F811
-                                               lossless_prio_list,         # noqa: F811
-                                               all_prio_list,        # noqa: F811
-                                               get_snappi_ports,         # noqa: F811
-                                               tbinfo,              # noqa: F811
-                                               setup_ports_and_dut,          # noqa: F811
-                                               disable_pfcwd,                # noqa: F811
-                                               reboot_duts):                 # noqa: F811
+                                               prio_dscp_map,               # noqa: F811
+                                               lossless_prio_list,          # noqa: F811
+                                               all_prio_list,               # noqa: F811
+                                               get_snappi_ports,            # noqa: F811
+                                               tbinfo,                      # noqa: F811
+                                               reboot_duts_and_disable_wd,  # noqa: F811
+                                               setup_ports_and_dut):        # noqa: F811
     """
     Test if PFC can pause a single lossless priority even after various types of reboot in multidut setup
 
@@ -226,7 +226,8 @@ def test_pfc_pause_single_lossless_prio_reboot(snappi_api,                   # n
         prio_dscp_map (pytest fixture): priority vs. DSCP map (key = priority).
         lossless_prio_list (pytest fixture): list of all the lossless priorities
         tbinfo (pytest fixture): fixture provides information about testbed
-        get_snappi_ports (pytest fixture): gets snappi ports and connected DUT port info and returns as a list
+        reboot_duts_and_disable_wd (pytest fixture): fixture reboots associated DUTs and disables Credit and PFC WD
+        setup_ports_and_dut (pytest fixture): gets snappi ports and connected DUT port info and returns as a list
     Returns:
         N/A
     """
@@ -268,9 +269,8 @@ def test_pfc_pause_multi_lossless_prio_reboot(snappi_api,                   # no
                                               lossless_prio_list,           # noqa: F811
                                               get_snappi_ports,             # noqa: F811
                                               tbinfo,                       # noqa: F811
-                                              setup_ports_and_dut,          # noqa: F811
-                                              disable_pfcwd,                # noqa: F811
-                                              reboot_duts):                 # noqa: F811
+                                              reboot_duts_and_disable_wd,   # noqa: F811
+                                              setup_ports_and_dut):         # noqa: F811
     """
     Test if PFC can pause multiple lossless priorities even after various types of reboot in multidut setup
 
@@ -284,7 +284,8 @@ def test_pfc_pause_multi_lossless_prio_reboot(snappi_api,                   # no
         lossless_prio_list (pytest fixture): list of all the lossless priorities
         lossy_prio_list (pytest fixture): list of all the lossy priorities
         tbinfo (pytest fixture): fixture provides information about testbed
-        get_snappi_ports (pytest fixture): gets snappi ports and connected DUT port info and returns as a list
+        reboot_duts_and_disable_wd (pytest fixture): fixture reboots associated DUTs and disables Credit and PFC WD
+        setup_ports_and_dut (pytest fixture): gets snappi ports and connected DUT port info and returns as a list
     Returns:
         N/A
     """