New test case for OQ watchdog

[zhixzhu]

Description of PR

Summary:
Fixes # (issue)
New test case for OQ watchdog.

To trigger OQ watchdog, it requires some OQ(s) are stuck, and all other OQ(s) of the same port are idle(no enqueue, no dequeue, queue empty).
Queue 7 has some control packets transmitting from time to time, so added some additional test steps to make OQ7 in idle state.

Test steps:
1. block voq7, sys_port scheduler set Q7 credit_pir to 0
2. fill leakout of Q7 by ping, make sure no packet dequeue/enqueue in OQ7 afterwards
3. block oq0, sys_port scheduler set Q0 transmit_pir to 0
4. send traffic on Q0, oq watchdog should be triggered in about 5 seconds
5. Unblock voq7 and oq0 to restore the system state
6. Run TrafficSanityTest to verify the system state is restored

Type of change

• Bug fix
• Testbed and Framework(new/improvement)
• New Test case
  • Skipped for non-supported platforms
• Test case improvement

Back port request

• 202205
• 202305
• 202311
• 202405
• 202411
• 202505

Approach

What is the motivation for this PR?

To verify OQ watchdog.

How did you do it?

Add a new test case to verify OQ watchdog.

How did you verify/test it?

Ran it on testbed.

T1 c-mono 8101-O8C48:

INFO:root:Can not get Allure report URL. Please check logs
---------------------------------------------------------------- live log sessionfinish -----------------------------------------------------------------
21:32:09 __init__.pytest_terminal_summary         L0067 INFO   | Can not get Allure report URL. Please check logs
================================================================ short test summary info ================================================================
PASSED qos/test_oq_watchdog.py::TestOqWatchdog::testOqWatchdog[single_asic]
======================================================= 1 passed, 1 warning in 274.62s (0:04:34) ========================================================
sonic@sonic-ucs-m6-26:/data/tests$ 

------------------------------- generated xml file: /tmp/qos/test_voq_watchdog_2025-06-11-23-33-53.xml --------------------------------
INFO:root:Can not get Allure report URL. Please check logs
------------------------------------------------------- live log sessionfinish --------------------------------------------------------
23:41:23 __init__.pytest_terminal_summary         L0067 INFO   | Can not get Allure report URL. Please check logs
======================================================= short test summary info =======================================================
PASSED qos/test_voq_watchdog.py::TestVoqWatchdog::testQosSaiVoqWatchdog[single_asic]
PASSED qos/test_voq_watchdog.py::TestVoqWatchdog::testQosSaiVoqWatchdogDisable[single_asic]
============================================== 2 passed, 1 warning in 449.32s (0:07:29) ===============================================
sonic@sonic-ucs-m6-26:/data/tests$ 

---------------------------------- generated xml file: /tmp/qos/test_qos_sai_2025-06-12-23-53-38.xml ----------------------------------
INFO:root:Can not get Allure report URL. Please check logs
------------------------------------------------------- live log sessionfinish --------------------------------------------------------
00:50:14 __init__.pytest_terminal_summary         L0067 INFO   | Can not get Allure report URL. Please check logs
======================================================= short test summary info =======================================================
PASSED qos/test_qos_sai.py::TestQosSai::testParameter[single_asic]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiPfcXoffLimit[single_asic-xoff_1]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiPfcXoffLimit[single_asic-xoff_2]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiPfcXonLimit[single_asic-xon_1]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiPfcXonLimit[single_asic-xon_2]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiLosslessVoq[single_asic-lossless_voq_1]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiLosslessVoq[single_asic-lossless_voq_2]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiLosslessVoq[single_asic-lossless_voq_4]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiSharedReservationSize[single_asic-shared_res_size_1]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiSharedReservationSize[single_asic-shared_res_size_2]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiBufferPoolWatermark[single_asic-wm_buf_pool_lossy]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiLossyQueue[single_asic]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiLossyQueueVoq[single_asic-lossy_queue_voq_1]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiDscpQueueMapping[single_asic]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiDwrr[single_asic]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiPgSharedWatermark[single_asic-wm_pg_shared_lossless]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiPgSharedWatermark[single_asic-wm_pg_shared_lossy]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiPGDrop[single_asic]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiQSharedWatermark[single_asic-wm_q_shared_lossy]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiDscpToPgMapping[single_asic]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiDwrrWeightChange[single_asic]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiQWatermarkAllPorts[single_asic-wm_q_wm_all_ports]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiLossyQueueVoqMultiSrc[single_asic]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiXonHysteresis[single_asic-xon_hysteresis_1]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiXonHysteresis[single_asic-xon_hysteresis_2]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiXonHysteresis[single_asic-xon_hysteresis_3]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiXonHysteresis[single_asic-xon_hysteresis_4]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiXonHysteresis[single_asic-xon_hysteresis_5]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiXonHysteresis[single_asic-xon_hysteresis_6]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiXonHysteresis[single_asic-xon_hysteresis_7]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiXonHysteresis[single_asic-xon_hysteresis_8]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiXonHysteresis[single_asic-xon_hysteresis_9]

T2:

----------------------------- generated xml file: /run_logs/qos/test_oq_watchdog_2025-06-16-01-05-06.xml ------------------------------
INFO:root:Can not get Allure report URL. Please check logs
------------------------------------------------------- live log sessionfinish --------------------------------------------------------
02:12:25 __init__.pytest_terminal_summary         L0067 INFO   | Can not get Allure report URL. Please check logs
======================================================= short test summary info =======================================================
PASSED qos/test_oq_watchdog.py::TestOqWatchdog::testOqWatchdog[single_asic]
PASSED qos/test_oq_watchdog.py::TestOqWatchdog::testOqWatchdog[single_dut_multi_asic]
PASSED qos/test_oq_watchdog.py::TestOqWatchdog::testOqWatchdog[multi_dut_longlink_to_shortlink]
PASSED qos/test_oq_watchdog.py::TestOqWatchdog::testOqWatchdog[multi_dut_shortlink_to_shortlink]
PASSED qos/test_oq_watchdog.py::TestOqWatchdog::testOqWatchdog[multi_dut_shortlink_to_longlink]
============================================== 5 passed, 1 warning in 4037.63s (1:07:17) ==============================================
sonic@sonic-ucs-m6-11:/data/tests$ 

----------------------------- generated xml file: /run_logs/qos/test_voq_watchdog_2025-06-16-05-42-48.xml -----------------------------
INFO:root:Can not get Allure report URL. Please check logs
------------------------------------------------------- live log sessionfinish --------------------------------------------------------
07:14:11 __init__.pytest_terminal_summary         L0067 INFO   | Can not get Allure report URL. Please check logs
======================================================= short test summary info =======================================================
PASSED qos/test_voq_watchdog.py::TestVoqWatchdog::testVoqWatchdog[single_asic]
PASSED qos/test_voq_watchdog.py::TestVoqWatchdog::testVoqWatchdogDisable[single_asic]
PASSED qos/test_voq_watchdog.py::TestVoqWatchdog::testVoqWatchdog[single_dut_multi_asic]
PASSED qos/test_voq_watchdog.py::TestVoqWatchdog::testVoqWatchdogDisable[single_dut_multi_asic]
PASSED qos/test_voq_watchdog.py::TestVoqWatchdog::testVoqWatchdog[multi_dut_longlink_to_shortlink]
PASSED qos/test_voq_watchdog.py::TestVoqWatchdog::testVoqWatchdogDisable[multi_dut_longlink_to_shortlink]
PASSED qos/test_voq_watchdog.py::TestVoqWatchdog::testVoqWatchdog[multi_dut_shortlink_to_shortlink]
PASSED qos/test_voq_watchdog.py::TestVoqWatchdog::testVoqWatchdogDisable[multi_dut_shortlink_to_shortlink]
PASSED qos/test_voq_watchdog.py::TestVoqWatchdog::testVoqWatchdog[multi_dut_shortlink_to_longlink]
PASSED qos/test_voq_watchdog.py::TestVoqWatchdog::testVoqWatchdogDisable[multi_dut_shortlink_to_longlink]

------------------------------- generated xml file: /run_logs/qos/test_qos_sai_2025-06-16-16-45-17.xml --------------------------------
INFO:root:Can not get Allure report URL. Please check logs
------------------------------------------------------- live log sessionfinish --------------------------------------------------------
20:41:40 __init__.pytest_terminal_summary         L0067 INFO   | Can not get Allure report URL. Please check logs
======================================================= short test summary info =======================================================
PASSED qos/test_qos_sai.py::TestQosSai::testParameter[single_asic]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiPfcXoffLimit[single_asic-xoff_1]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiPfcXoffLimit[single_asic-xoff_2]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiPfcXonLimit[single_asic-xon_1]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiPfcXonLimit[single_asic-xon_2]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiLosslessVoq[single_asic-lossless_voq_1]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiLosslessVoq[single_asic-lossless_voq_2]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiLosslessVoq[single_asic-lossless_voq_3]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiLosslessVoq[single_asic-lossless_voq_4]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiBufferPoolWatermark[single_asic-wm_buf_pool_lossy]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiLossyQueue[single_asic]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiDscpQueueMapping[single_asic]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiDwrr[single_asic]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiPgSharedWatermark[single_asic-wm_pg_shared_lossless]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiPgSharedWatermark[single_asic-wm_pg_shared_lossy]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiPGDrop[single_asic]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiQSharedWatermark[single_asic-wm_q_shared_lossless]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiQSharedWatermark[single_asic-wm_q_shared_lossy]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiDscpToPgMapping[single_asic]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiDwrrWeightChange[single_asic]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiQWatermarkAllPorts[single_asic-wm_q_wm_all_ports]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiLossyQueueVoqMultiSrc[single_asic]
PASSED qos/test_qos_sai.py::TestQosSai::testParameter[single_dut_multi_asic]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiPfcXoffLimit[single_dut_multi_asic-xoff_1]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiPfcXoffLimit[single_dut_multi_asic-xoff_2]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiPfcXonLimit[single_dut_multi_asic-xon_1]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiPfcXonLimit[single_dut_multi_asic-xon_2]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiBufferPoolWatermark[single_dut_multi_asic-wm_buf_pool_lossless]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiBufferPoolWatermark[single_dut_multi_asic-wm_buf_pool_lossy]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiLossyQueue[single_dut_multi_asic]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiDscpQueueMapping[single_dut_multi_asic]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiDwrr[single_dut_multi_asic]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiPgSharedWatermark[single_dut_multi_asic-wm_pg_shared_lossless]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiPGDrop[single_dut_multi_asic]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiQSharedWatermark[single_dut_multi_asic-wm_q_shared_lossless]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiQSharedWatermark[single_dut_multi_asic-wm_q_shared_lossy]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiDscpToPgMapping[single_dut_multi_asic]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiDwrrWeightChange[single_dut_multi_asic]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiQWatermarkAllPorts[single_dut_multi_asic-wm_q_wm_all_ports]
PASSED qos/test_qos_sai.py::TestQosSai::testParameter[multi_dut_longlink_to_shortlink]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiPfcXoffLimit[multi_dut_longlink_to_shortlink-xoff_1]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiPfcXoffLimit[multi_dut_longlink_to_shortlink-xoff_2]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiPfcXonLimit[multi_dut_longlink_to_shortlink-xon_1]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiPfcXonLimit[multi_dut_longlink_to_shortlink-xon_2]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiBufferPoolWatermark[multi_dut_longlink_to_shortlink-wm_buf_pool_lossy]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiLossyQueue[multi_dut_longlink_to_shortlink]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiDscpQueueMapping[multi_dut_longlink_to_shortlink]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiDwrr[multi_dut_longlink_to_shortlink]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiQSharedWatermark[multi_dut_longlink_to_shortlink-wm_q_shared_lossy]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiDscpToPgMapping[multi_dut_longlink_to_shortlink]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiDwrrWeightChange[multi_dut_longlink_to_shortlink]
PASSED qos/test_qos_sai.py::TestQosSai::testParameter[multi_dut_shortlink_to_shortlink]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiPfcXoffLimit[multi_dut_shortlink_to_shortlink-xoff_1]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiPfcXoffLimit[multi_dut_shortlink_to_shortlink-xoff_2]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiPfcXonLimit[multi_dut_shortlink_to_shortlink-xon_1]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiPfcXonLimit[multi_dut_shortlink_to_shortlink-xon_2]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiBufferPoolWatermark[multi_dut_shortlink_to_shortlink-wm_buf_pool_lossless]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiBufferPoolWatermark[multi_dut_shortlink_to_shortlink-wm_buf_pool_lossy]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiLossyQueue[multi_dut_shortlink_to_shortlink]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiDscpQueueMapping[multi_dut_shortlink_to_shortlink]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiDwrr[multi_dut_shortlink_to_shortlink]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiPgSharedWatermark[multi_dut_shortlink_to_shortlink-wm_pg_shared_lossless]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiPGDrop[multi_dut_shortlink_to_shortlink]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiQSharedWatermark[multi_dut_shortlink_to_shortlink-wm_q_shared_lossless]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiQSharedWatermark[multi_dut_shortlink_to_shortlink-wm_q_shared_lossy]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiDscpToPgMapping[multi_dut_shortlink_to_shortlink]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiDwrrWeightChange[multi_dut_shortlink_to_shortlink]
PASSED qos/test_qos_sai.py::TestQosSai::testParameter[multi_dut_shortlink_to_longlink]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiPfcXoffLimit[multi_dut_shortlink_to_longlink-xoff_1]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiPfcXoffLimit[multi_dut_shortlink_to_longlink-xoff_2]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiPfcXonLimit[multi_dut_shortlink_to_longlink-xon_1]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiPfcXonLimit[multi_dut_shortlink_to_longlink-xon_2]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiBufferPoolWatermark[multi_dut_shortlink_to_longlink-wm_buf_pool_lossy]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiLossyQueue[multi_dut_shortlink_to_longlink]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiDscpQueueMapping[multi_dut_shortlink_to_longlink]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiDwrr[multi_dut_shortlink_to_longlink]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiPgSharedWatermark[multi_dut_shortlink_to_longlink-wm_pg_shared_lossless]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiPGDrop[multi_dut_shortlink_to_longlink]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiQSharedWatermark[multi_dut_shortlink_to_longlink-wm_q_shared_lossy]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiDscpToPgMapping[multi_dut_shortlink_to_longlink]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiDwrrWeightChange[multi_dut_shortlink_to_longlink]

Any platform specific information?

Only run on platforms which enable OQ watchdog.

Supported testbed topology if it's a new test case?

T0, T1, and T2.

Documentation

[mssonicbld]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[mssonicbld]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[mssonicbld]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[mssonicbld]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[zhixzhu]

Failure of Azure.sonic-mgmt (Test impacted-area-kvmtest-multi-asic-t1 by Elastictest - optional):
https://elastictest.org/scheduler/publictestplan/684a1631b96fef74050c266d
Lock testbeds failed with error: Request timed out. Retries due to rate limiting: True., full error: {'ok': 0.0, 'errmsg': 'Request timed out. Retries due to rate limiting: True.', 'code': 50, 'codeName': 'ExceededTimeLimit'}

[mssonicbld]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[mssonicbld]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[zhixzhu]

@sdszhang please help review.

[sdszhang]

lgtm

[sdszhang]

This feature is enabled in msft-202405 image. Will need to include this in msft-202405 sonic-mgmt branch.

[mssonicbld]

Cherry-pick PR to 202505: #19111

[zhixzhu]

Hi @zhixzhu , please fix the conflict the raise PR to msft-202405 branch

@yejianquan PR to double commit to msft-202405 branch: Azure/sonic-mgmt.msft#433

[mssonicbld]

@zhixzhu PR conflicts with 202411 branch

[zhixzhu]

@zhixzhu PR conflicts with 202411 branch

Double commit to 202411 branch: #20610