Porting 15718 to 202405: Adding a fixture to set scheduler to slower speeds and revert it back. by rraghav-cisco · Pull Request #16199 · sonic-net/sonic-mgmt

rraghav-cisco · 2024-12-23T05:13:43Z

Description of PR

Summary:
Fixes the flakiness of DWRR testcase. The PR adds a new fixture that slows down the scheduler without changing the underlying algorithm. This allows the dWRR test to pass consitently.

Type of change

Bug fix
Testbed and Framework(new/improvement)
Test case(new/improvement)

Back port request

Approach

What is the motivation for this PR?

How did you do it?

How did you verify/test it?

Any platform specific information?

Supported testbed topology if it's a new test case?

Documentation

mssonicbld · 2024-12-23T05:13:46Z

/azp run

azure-pipelines · 2024-12-23T05:13:56Z

Azure Pipelines successfully started running 1 pipeline(s).

yejianquan · 2024-12-24T00:08:09Z

Hi @rraghav-cisco , could you please paste the test result with the code that resolved the conlficts?
Thanks!

rraghav-cisco · 2024-12-24T00:29:41Z

Hi @rraghav-cisco , could you please paste the test result with the code that resolved the conlficts? Thanks!

Hi @yejianquan , I am seeing pass in Dwrr, but fails in DwrrWeightChange:

___________________________________________________________________________________________ TestQosSai.testQosSaiDwrr[single_asic] ___________________________________________________________________________________________
______________________________________________________________________________________ TestQosSai.testQosSaiDwrr[single_dut_multi_asic] ______________________________________________________________________________________
_________________________________________________________________________________ TestQosSai.testQosSaiDwrr[multi_dut_longlink_to_shortlink] _________________________________________________________________________________
_________________________________________________________________________________ TestQosSai.testQosSaiDwrr[multi_dut_shortlink_to_longlink] _________________________________________________________________________________
-------------------------------------------------------- generated xml file: /run_logs/19351-dwrr-only/2024-12-20-16-14-07/2/qos/test_qos_sai_2024-12-20-17-03-53.xml --------------------------------------------------------
INFO:root:Can not get Allure report URL. Please check logs
--------------------------------------------------------------------------------------------------- live log sessionfinish ---------------------------------------------------------------------------------------------------
17:53:46 __init__.pytest_terminal_summary         L0067 INFO   | Can not get Allure report URL. Please check logs
================================================================================================== short test summary info ===================================================================================================
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiDwrr[single_asic]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiDwrr[single_dut_multi_asic]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiDwrr[multi_dut_longlink_to_shortlink]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiDwrr[multi_dut_shortlink_to_longlink]
SKIPPED [1] qos/test_qos_sai.py:1541: Don't have 2 shortlink frontend nodes - so can't run multi_dut_shortlink_to_shortlinktests
SKIPPED [1] qos/test_qos_sai.py:2091: Don't have 2 shortlink frontend nodes - so can't run multi_dut_shortlink_to_shortlinktests
FAILED qos/test_qos_sai.py::TestQosSai::testQosSaiDwrrWeightChange[single_asic] - tests.common.errors.RunAnsibleModuleFail: run module shell failed, Ansible Results =>
FAILED qos/test_qos_sai.py::TestQosSai::testQosSaiDwrrWeightChange[single_dut_multi_asic] - tests.common.errors.RunAnsibleModuleFail: run module shell failed, Ansible Results =>
FAILED qos/test_qos_sai.py::TestQosSai::testQosSaiDwrrWeightChange[multi_dut_longlink_to_shortlink] - tests.common.errors.RunAnsibleModuleFail: run module shell failed, Ansible Results =>
FAILED qos/test_qos_sai.py::TestQosSai::testQosSaiDwrrWeightChange[multi_dut_shortlink_to_longlink] - tests.common.errors.RunAnsibleModuleFail: run module shell failed, Ansible Results =>
======================================================================= 4 failed, 4 passed, 2 skipped, 235 deselected, 1 warning in 2991.24s (0:49:51) =======================================================================
sonic@202405-qos-sonic-mgmt-prod:/data/tests$

yejianquan · 2024-12-24T03:56:47Z

Hi @rraghav-cisco , could you please paste the test result with the code that resolved the conlficts? Thanks!

Hi @yejianquan , I am seeing pass in Dwrr, but fails in DwrrWeightChange:

___________________________________________________________________________________________ TestQosSai.testQosSaiDwrr[single_asic] ___________________________________________________________________________________________
______________________________________________________________________________________ TestQosSai.testQosSaiDwrr[single_dut_multi_asic] ______________________________________________________________________________________
_________________________________________________________________________________ TestQosSai.testQosSaiDwrr[multi_dut_longlink_to_shortlink] _________________________________________________________________________________
_________________________________________________________________________________ TestQosSai.testQosSaiDwrr[multi_dut_shortlink_to_longlink] _________________________________________________________________________________
-------------------------------------------------------- generated xml file: /run_logs/19351-dwrr-only/2024-12-20-16-14-07/2/qos/test_qos_sai_2024-12-20-17-03-53.xml --------------------------------------------------------
INFO:root:Can not get Allure report URL. Please check logs
--------------------------------------------------------------------------------------------------- live log sessionfinish ---------------------------------------------------------------------------------------------------
17:53:46 __init__.pytest_terminal_summary         L0067 INFO   | Can not get Allure report URL. Please check logs
================================================================================================== short test summary info ===================================================================================================
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiDwrr[single_asic]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiDwrr[single_dut_multi_asic]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiDwrr[multi_dut_longlink_to_shortlink]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiDwrr[multi_dut_shortlink_to_longlink]
SKIPPED [1] qos/test_qos_sai.py:1541: Don't have 2 shortlink frontend nodes - so can't run multi_dut_shortlink_to_shortlinktests
SKIPPED [1] qos/test_qos_sai.py:2091: Don't have 2 shortlink frontend nodes - so can't run multi_dut_shortlink_to_shortlinktests
FAILED qos/test_qos_sai.py::TestQosSai::testQosSaiDwrrWeightChange[single_asic] - tests.common.errors.RunAnsibleModuleFail: run module shell failed, Ansible Results =>
FAILED qos/test_qos_sai.py::TestQosSai::testQosSaiDwrrWeightChange[single_dut_multi_asic] - tests.common.errors.RunAnsibleModuleFail: run module shell failed, Ansible Results =>
FAILED qos/test_qos_sai.py::TestQosSai::testQosSaiDwrrWeightChange[multi_dut_longlink_to_shortlink] - tests.common.errors.RunAnsibleModuleFail: run module shell failed, Ansible Results =>
FAILED qos/test_qos_sai.py::TestQosSai::testQosSaiDwrrWeightChange[multi_dut_shortlink_to_longlink] - tests.common.errors.RunAnsibleModuleFail: run module shell failed, Ansible Results =>
======================================================================= 4 failed, 4 passed, 2 skipped, 235 deselected, 1 warning in 2991.24s (0:49:51) =======================================================================
sonic@202405-qos-sonic-mgmt-prod:/data/tests$

@rraghav-cisco is DwrrWeightChange because of the testbed issue or the script?

sdszhang

LGTM

sdszhang · 2025-01-03T00:56:12Z

Hi @rraghav-cisco , this new test failed on T0/T1 setup #16314. could you please fix it ASAP?

yejianquan · 2025-01-03T01:01:59Z

convert to draft because it fails on t0/t1 cisco devices #16314

sdszhang

pls fix the error on t0/t1 as well.

sdszhang · 2025-01-03T03:01:03Z

@rraghav-cisco can you include fix #16315 in this PR?

mssonicbld · 2025-01-03T03:13:42Z

/azp run

azure-pipelines · 2025-01-03T03:13:53Z

Azure Pipelines successfully started running 1 pipeline(s).

rraghav-cisco · 2025-01-03T03:13:59Z

@rraghav-cisco can you include fix #16315 in this PR?

@sdszhang , done.

rraghav-cisco · 2025-01-06T18:18:31Z

Hi @rraghav-cisco , could you please paste the test result with the code that resolved the conlficts? Thanks!

Hi @yejianquan , I am seeing pass in Dwrr, but fails in DwrrWeightChange:

___________________________________________________________________________________________ TestQosSai.testQosSaiDwrr[single_asic] ___________________________________________________________________________________________
______________________________________________________________________________________ TestQosSai.testQosSaiDwrr[single_dut_multi_asic] ______________________________________________________________________________________
_________________________________________________________________________________ TestQosSai.testQosSaiDwrr[multi_dut_longlink_to_shortlink] _________________________________________________________________________________
_________________________________________________________________________________ TestQosSai.testQosSaiDwrr[multi_dut_shortlink_to_longlink] _________________________________________________________________________________
-------------------------------------------------------- generated xml file: /run_logs/19351-dwrr-only/2024-12-20-16-14-07/2/qos/test_qos_sai_2024-12-20-17-03-53.xml --------------------------------------------------------
INFO:root:Can not get Allure report URL. Please check logs
--------------------------------------------------------------------------------------------------- live log sessionfinish ---------------------------------------------------------------------------------------------------
17:53:46 __init__.pytest_terminal_summary         L0067 INFO   | Can not get Allure report URL. Please check logs
================================================================================================== short test summary info ===================================================================================================
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiDwrr[single_asic]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiDwrr[single_dut_multi_asic]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiDwrr[multi_dut_longlink_to_shortlink]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiDwrr[multi_dut_shortlink_to_longlink]
SKIPPED [1] qos/test_qos_sai.py:1541: Don't have 2 shortlink frontend nodes - so can't run multi_dut_shortlink_to_shortlinktests
SKIPPED [1] qos/test_qos_sai.py:2091: Don't have 2 shortlink frontend nodes - so can't run multi_dut_shortlink_to_shortlinktests
FAILED qos/test_qos_sai.py::TestQosSai::testQosSaiDwrrWeightChange[single_asic] - tests.common.errors.RunAnsibleModuleFail: run module shell failed, Ansible Results =>
FAILED qos/test_qos_sai.py::TestQosSai::testQosSaiDwrrWeightChange[single_dut_multi_asic] - tests.common.errors.RunAnsibleModuleFail: run module shell failed, Ansible Results =>
FAILED qos/test_qos_sai.py::TestQosSai::testQosSaiDwrrWeightChange[multi_dut_longlink_to_shortlink] - tests.common.errors.RunAnsibleModuleFail: run module shell failed, Ansible Results =>
FAILED qos/test_qos_sai.py::TestQosSai::testQosSaiDwrrWeightChange[multi_dut_shortlink_to_longlink] - tests.common.errors.RunAnsibleModuleFail: run module shell failed, Ansible Results =>
======================================================================= 4 failed, 4 passed, 2 skipped, 235 deselected, 1 warning in 2991.24s (0:49:51) =======================================================================
sonic@202405-qos-sonic-mgmt-prod:/data/tests$

@rraghav-cisco is DwrrWeightChange because of the testbed issue or the script?

@yejianquan : The issue was in my workspace. After fixing the workspace, all of these passed.

mssonicbld · 2025-01-06T20:16:55Z

/azp run

azure-pipelines · 2025-01-06T20:17:06Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2025-01-09T04:04:28Z

/azp run

azure-pipelines · 2025-01-09T04:04:40Z

Azure Pipelines successfully started running 1 pipeline(s).

sdszhang · 2025-01-09T06:26:11Z

@kevinskwang @XuChen-MSFT for viz. This is the manual cherry-pick PR for #15718 and #16315 combined into 202405.

mssonicbld · 2025-01-09T15:03:59Z

/azp run

azure-pipelines · 2025-01-09T15:04:12Z

Azure Pipelines successfully started running 1 pipeline(s).

Description of PR Optimize the qos/test_qos_sai.py test to reduce the running time. Summary: Fixes # (issue) Microsoft ADO 30056122 Type of change Bug fix Testbed and Framework(new/improvement) New Test case Skipped for non-supported platforms Add ownership here(Microsft required only) Test case improvement Approach What is the motivation for this PR? The running time of the qos/test_qos_sai.py test is too long (~9h) on T2 chassis so we wanted to reduce the running time. With this implementation, the running time will be reduced to (~7.5h) How did you do it? How did you verify/test it? I ran the updated code and can confirm it's working well on T2 chassis: Elastictest link. The 2 DWRR failures are expected, which will be fixed after #16199 I also ran a T1 regression test to confirm: Elastictest link co-authorized by: jianquanye@microsoft.com

Description of PR Optimize the qos/test_qos_sai.py test to reduce the running time. Summary: Fixes # (issue) Microsoft ADO 30056122 Type of change Bug fix Testbed and Framework(new/improvement) New Test case Skipped for non-supported platforms Add ownership here(Microsft required only) Test case improvement Approach What is the motivation for this PR? The running time of the qos/test_qos_sai.py test is too long (~9h) on T2 chassis so we wanted to reduce the running time. With this implementation, the running time will be reduced to (~7.5h) How did you do it? How did you verify/test it? I ran the updated code and can confirm it's working well on T2 chassis: Elastictest link. The 2 DWRR failures are expected, which will be fixed after sonic-net#16199 I also ran a T1 regression test to confirm: Elastictest link co-authorized by: jianquanye@microsoft.com

sdszhang · 2025-01-13T04:03:05Z

Test result
T0/T1 PASSED: T0/T1 result
T2 PASSED: T2 result

Description of PR Optimize the qos/test_qos_sai.py test to reduce the running time. Summary: Fixes # (issue) Microsoft ADO 30056122 Type of change Bug fix Testbed and Framework(new/improvement) New Test case Skipped for non-supported platforms Add ownership here(Microsft required only) Test case improvement Approach What is the motivation for this PR? The running time of the qos/test_qos_sai.py test is too long (~9h) on T2 chassis so we wanted to reduce the running time. With this implementation, the running time will be reduced to (~7.5h) How did you do it? How did you verify/test it? I ran the updated code and can confirm it's working well on T2 chassis: Elastictest link. The 2 DWRR failures are expected, which will be fixed after sonic-net#16199 I also ran a T1 regression test to confirm: Elastictest link co-authorized by: jianquanye@microsoft.com

Description of PR Optimize the qos/test_qos_sai.py test to reduce the running time. Summary: Fixes # (issue) Microsoft ADO 30056122 Type of change Bug fix Testbed and Framework(new/improvement) New Test case Skipped for non-supported platforms Add ownership here(Microsft required only) Test case improvement Approach What is the motivation for this PR? The running time of the qos/test_qos_sai.py test is too long (~9h) on T2 chassis so we wanted to reduce the running time. With this implementation, the running time will be reduced to (~7.5h) How did you do it? How did you verify/test it? I ran the updated code and can confirm it's working well on T2 chassis: Elastictest link. The 2 DWRR failures are expected, which will be fixed after #16199 I also ran a T1 regression test to confirm: Elastictest link co-authorized by: jianquanye@microsoft.com

Description of PR Optimize the qos/test_qos_sai.py test to reduce the running time. Summary: Fixes # (issue) Microsoft ADO 30056122 Type of change Bug fix Testbed and Framework(new/improvement) New Test case Skipped for non-supported platforms Add ownership here(Microsft required only) Test case improvement Approach What is the motivation for this PR? The running time of the qos/test_qos_sai.py test is too long (~9h) on T2 chassis so we wanted to reduce the running time. With this implementation, the running time will be reduced to (~7.5h) How did you do it? How did you verify/test it? I ran the updated code and can confirm it's working well on T2 chassis: Elastictest link. The 2 DWRR failures are expected, which will be fixed after sonic-net#16199 I also ran a T1 regression test to confirm: Elastictest link co-authorized by: jianquanye@microsoft.com

Porting 15718 to 202405.

13374fe

rraghav-cisco mentioned this pull request Dec 23, 2024

Adding a fixture to set scheduler to slower speeds and revert it back. #15718

Merged

8 tasks

sdszhang self-requested a review January 2, 2025 04:29

sdszhang approved these changes Jan 3, 2025

View reviewed changes

sdszhang mentioned this pull request Jan 3, 2025

[Bug]: dwrr failed on T0/T1 cisco platform. #16314

Closed

yejianquan marked this pull request as draft January 3, 2025 01:01

sdszhang reviewed Jan 3, 2025

View reviewed changes

Include the single-asic fix from PR:16315 in the same PR.

ea5888b

Fix the ptf code to handle single and mAsic combos.

9a28bb0

Adding the new logic from PR:16315 to this PR.

828e97e

rraghav-cisco marked this pull request as ready for review January 9, 2025 04:23

sdszhang added the chassis label Jan 9, 2025

Change the name of the function, remove '_and_run' part.

3b58f42

cyw233 mentioned this pull request Jan 10, 2025

refactor: optimize qos sai test #16428

Merged

12 tasks

cyw233 mentioned this pull request Jan 13, 2025

[manual] [PR:16428] refactor: optimize qos sai test #16472

Merged

12 tasks

cyw233 mentioned this pull request Jan 13, 2025

[manual] [PR:16428] refactor: optimize qos sai test #16474

Closed

12 tasks

yejianquan merged commit 942b849 into sonic-net:202405 Jan 13, 2025

rraghav-cisco mentioned this pull request Jan 13, 2025

qos-sai:dwrr:cisco-8000:Handle non-multiasic part as well for the dshell-script change. #16315

Merged

9 tasks

mssonicbld mentioned this pull request Jan 13, 2025

[action] [PR:16428] refactor: optimize qos sai test #16488

Merged

12 tasks

Conversation

rraghav-cisco commented Dec 23, 2024

Description of PR

Type of change

Back port request

Approach

What is the motivation for this PR?

How did you do it?

How did you verify/test it?

Any platform specific information?

Supported testbed topology if it's a new test case?

Documentation

Uh oh!

mssonicbld commented Dec 23, 2024

Uh oh!

azure-pipelines bot commented Dec 23, 2024

Uh oh!

yejianquan commented Dec 24, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rraghav-cisco commented Dec 24, 2024

Uh oh!

yejianquan commented Dec 24, 2024

Uh oh!

sdszhang left a comment

Choose a reason for hiding this comment

Uh oh!

sdszhang commented Jan 3, 2025

Uh oh!

yejianquan commented Jan 3, 2025

Uh oh!

sdszhang left a comment

Choose a reason for hiding this comment

Uh oh!

sdszhang commented Jan 3, 2025

Uh oh!

mssonicbld commented Jan 3, 2025

Uh oh!

azure-pipelines bot commented Jan 3, 2025

Uh oh!

rraghav-cisco commented Jan 3, 2025

Uh oh!

rraghav-cisco commented Jan 6, 2025

Uh oh!

mssonicbld commented Jan 6, 2025

Uh oh!

azure-pipelines bot commented Jan 6, 2025

Uh oh!

mssonicbld commented Jan 9, 2025

Uh oh!

azure-pipelines bot commented Jan 9, 2025

Uh oh!

sdszhang commented Jan 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mssonicbld commented Jan 9, 2025

Uh oh!

azure-pipelines bot commented Jan 9, 2025

Uh oh!

sdszhang commented Jan 13, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

yejianquan commented Dec 24, 2024 •

edited

Loading

sdszhang commented Jan 9, 2025 •

edited

Loading