[Snappi] Modified pfcwd tests to add warm-up traffic by developfast · Pull Request #7712 · sonic-net/sonic-mgmt

developfast · 2023-03-10T01:34:10Z

Description of PR

Summary: During the pfc watch dog (pfcwd) basic tests, pfc watch dog was being triggered by fake alerts i.e. when it was not supposed to be triggered. This was causing unnecessary test failures. To curb this issue, warm-up data traffic is initially sent before any other traffic (pfc pause storm and data traffic) to prevent pfcwd fake alerts caused by non-incremented packet counters during the pfcwd polling interval. This PR is in addition to #7499.

Type of change

Bug fix
Testbed and Framework(new/improvement)
Test case(new/improvement)

Back port request

201911
202012
202205

Approach

What is the motivation for this PR?

Existing issue with certain switches where PFC watch dog was being triggered by fake alerts.

How did you do it?

Added warm-up data traffic before any other traffic (pfc pause storm and data traffic) to prevent pfcwd fake alerts caused by non-incremented packet counters during the pfcwd polling interval.

How did you verify/test it?

The related PFCWD tests pass on a lab switch.

Any platform specific information?

Supported testbed topology if it's a new test case?

Documentation

mssonicbld · 2023-03-10T01:35:13Z

The pre-commit check detected issues in the files touched by this pull request.
The detected issues may be old or new. For new issues, please try to fix them.

For old issues, it is not mandatory to fix them because they were not caused by this change. It is unfair to blame
author of this pull request. But if you can take extra effort to fix the old issues as well, that would be great!

Detailed pre-commit check results:
trim trailing whitespace.................................................Failed - hook id: trailing-whitespace - exit code: 1 - files were modified by this hook Fixing tests/snappi/pfcwd/files/pfcwd_burst_storm_helper.py Fixing tests/snappi/pfcwd/files/pfcwd_multi_node_helper.py Fixing tests/snappi/pfcwd/files/pfcwd_runtime_traffic_helper.py fix end of files.........................................................Passed check yaml...........................................(no files to check)Skipped check for added large files..............................................Passed check python ast.........................................................Passed flake8...................................................................Failed - hook id: flake8 - exit code: 1 tests/snappi/pfcwd/files/pfcwd_multi_node_helper.py:6:1: F401 'tests.common.fixtures.conn_graph_facts.conn_graph_facts' imported but unused tests/snappi/pfcwd/files/pfcwd_multi_node_helper.py:6:1: F401 'tests.common.fixtures.conn_graph_facts.fanout_graph_facts' imported but unused tests/snappi/pfcwd/files/pfcwd_multi_node_helper.py:93:64: E502 the backslash is redundant between brackets ... [truncated extra lines, please run pre-commit locally to view full check results]

To run the pre-commit checks locally, you can follow below steps:

Ensure that default python is python3. In sonic-mgmt docker container, default python is python2. You can run
the check by activating the python3 virtual environment in sonic-mgmt docker container or outside of sonic-mgmt
docker container.
Ensure that the pre-commit package is installed:

sudo pip install pre-commit

Go to repository root folder
Install the pre-commit hooks:

pre-commit install

Use pre-commit to check staged file:

pre-commit

Alternatively, you can check committed files using:

pre-commit run --from-ref <commit_id> --to-ref <commit_id>

neethajohn · 2023-03-10T19:32:23Z

LGTM. @baiwei0427, please take a look

baiwei0427 · 2023-03-12T01:14:16Z

tests/snappi/pfcwd/files/pfcwd_burst_storm_helper.py

    pause_flow_dur_sec = poll_interval_sec * 0.5
    pause_flow_gap_sec = burst_cycle_sec - pause_flow_dur_sec

+    """ Warm up traffic is initially sent before any other traffic to prevent pfcwd 


prevent pfcwd fake alerts due to idle links

baiwei0427 · 2023-03-12T01:30:33Z

tests/snappi/pfcwd/files/pfcwd_burst_storm_helper.py

        pause_pkt.pause_class_7.value = pause_time[7]

-        pause_flow_start_time = id * (pause_flow_dur_sec + pause_flow_gap_sec)
+        pause_flow_start_time = id * (pause_flow_dur_sec + pause_flow_gap_sec) + WARM_UP_TRAFFIC_DUR


Maybe we should introduce a new argument "pause_flow_start_delay_sec" for __gen_traffic function

The id value is generated within the function itself (as well as pause_flow_dur_sec + pause_flow_gap_sec), so I prefer to keep it like it is for now. We can rename the variable to pause_flow_start_delay_sec?

baiwei0427 · 2023-03-12T01:38:43Z

tests/snappi/pfcwd/files/pfcwd_multi_node_helper.py

+
+    __gen_data_flows(testbed_config=testbed_config,


Add comments here "Generate warm up traffic"

baiwei0427 · 2023-03-12T01:43:15Z

tests/snappi/pfcwd/files/pfcwd_burst_storm_helper.py

+    fake alerts caused by non-incremented packet counters during pfcwd detection periods"""
+    warm_up_traffic_dur_sec = WARM_UP_TRAFFIC_DUR
+    warm_up_traffic_delay_sec = 0
+


Instead of generating "warm up traffic", "data flows" and "PFC storm" in a function, how about using two functions __gen_data_flows and __gen_pause_flow to generate different flows, just like what we did in pfcwd_multi_node_helper.py?

I was thinking the same - let me have a look at code re-usability and move around some components

baiwei0427 · 2023-03-13T18:04:08Z

tests/snappi/pfcwd/files/pfcwd_multi_node_helper.py

                     asic_type=asic_type)


+def sec_to_nanosec(secs):


This function has been defined, implemented and used multiple times.

Will move to a separate helper function file

mssonicbld · 2023-03-14T01:38:52Z

The pre-commit check detected issues in the files touched by this pull request.
The detected issues may be old or new. For new issues, please try to fix them.

For old issues, it is not mandatory to fix them because they were not caused by this change. It is unfair to blame
author of this pull request. But if you can take extra effort to fix the old issues as well, that would be great!

Detailed pre-commit check results:
trim trailing whitespace.................................................Passed fix end of files.........................................................Passed check yaml...........................................(no files to check)Skipped check for added large files..............................................Passed check python ast.........................................................Passed flake8...................................................................Failed - hook id: flake8 - exit code: 1 tests/snappi/pfcwd/files/pfcwd_basic_helper.py:6:1: F401 'tests.common.fixtures.conn_graph_facts.conn_graph_facts' imported but unused tests/snappi/pfcwd/files/pfcwd_basic_helper.py:6:1: F401 'tests.common.fixtures.conn_graph_facts.fanout_graph_facts' imported but unused tests/snappi/pfcwd/files/pfcwd_basic_helper.py:26:1: E302 expected 2 blank lines, found 1 tests/snappi/pfcwd/files/pfcwd_basic_helper.py:90:32: E225 missing whitespace around operator tests/snappi/pfcwd/files/pfcwd_basic_helper.py:359:23: E122 continuation line missing indentation or outdented tests/snappi/pfcwd/files/pfcwd_multi_node_helper.py:6:1: F401 'tests.common.fixtures.conn_graph_facts.conn_graph_facts' imported but unused tests/snappi/pfcwd/files/pfcwd_multi_node_helper.py:6:1: F401 'tests.common.fixtures.conn_graph_facts.fanout_graph_facts' imported but unused tests/snappi/pfcwd/files/pfcwd_multi_node_helper.py:93:64: E502 the backslash is redundant between brackets tests/snappi/pfcwd/files/pfcwd_multi_node_helper.py:94:52: E502 the backslash is redundant between brackets tests/snappi/pfcwd/files/pfcwd_multi_node_helper.py:97:60: E502 the backslash is redundant between brackets tests/snappi/pfcwd/files/pfcwd_multi_node_helper.py:98:50: E502 the backslash is redundant between brackets ... [truncated extra lines, please run pre-commit locally to view full check results]

To run the pre-commit checks locally, you can follow below steps:

Ensure that default python is python3. In sonic-mgmt docker container, default python is python2. You can run
the check by activating the python3 virtual environment in sonic-mgmt docker container or outside of sonic-mgmt
docker container.
Ensure that the pre-commit package is installed:

sudo pip install pre-commit

Go to repository root folder
Install the pre-commit hooks:

pre-commit install

Use pre-commit to check staged file:

pre-commit

Alternatively, you can check committed files using:

pre-commit run --from-ref <commit_id> --to-ref <commit_id>

* added warm-up traffic to all pfcwd tests

Signed-off-by: Austin Pham <austinpham@microsoft.com>

fix: migrate sonic-net#7370 to multidut code Signed-off-by: Austin Pham <austinpham@microsoft.com> fix: migrate sonic-net#9567 to multidut code Signed-off-by: Austin Pham <austinpham@microsoft.com> fix: migrate sonic-net#7367 to multidut code Signed-off-by: Austin Pham <austinpham@microsoft.com> fix: migrate sonic-net#7712 Signed-off-by: Austin Pham <austinpham@microsoft.com> fix: migrate sonic-net#7712 to pfcwd_multidut_runtime_traffic_helper Signed-off-by: Austin Pham <austinpham@microsoft.com> chore: update a2a parameterize Signed-off-by: Austin Pham <austinpham@microsoft.com>

Summary: Update the missing T0 T1 tests feature in multidut Fixes # (issue) 30929279 The following PR were migrated for pfcwd [Snappi] Increase number of calls for ARP wait #7370 [Snappi] PFCWD Tolerance Fixes #9567 [Snappi] Modifications to pfcwd a2a test design for mlnx 4600c #7367 to multidut code [Snappi] Modified pfcwd tests to add warm-up traffic #7712 Signed-off-by: Austin Pham <austinpham@microsoft.com>

Summary: Update the missing T0 T1 tests feature in multidut Fixes # (issue) 30929279 The following PR were migrated for pfcwd - sonic-net/sonic-mgmt#7370 - sonic-net/sonic-mgmt#9567 - sonic-net/sonic-mgmt#7367 to multidut code - sonic-net/sonic-mgmt#7712 ### Type of change  - [ ] Bug fix - [ ] Testbed and Framework(new/improvement) - [ ] New Test case - [ ] Skipped for non-supported platforms - [ ] Test case improvement ### Back port request - [ ] 202012 - [ ] 202205 - [ ] 202305 - [ ] 202311 - [ ] 202405 - [ ] 202411 ### Approach #### What is the motivation for this PR? #### How did you do it? #### How did you verify/test it? #### Any platform specific information? #### Supported testbed topology if it's a new test case? ### Documentation  snappi_tests/multidut/pfcwd/test_multidut_pfcwd_m2o_with_snappi.py ``` -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html - generated xml file: /var/src/sonic-mgmt_vmsvc5-t2-8800-ixia_669de75fbbfea63e8b73b319/tests/logs/snappi_tests/multidut/pfcwd/test_multidut_pfcwd_m2o_with_snappi.xml - ================== 4 passed, 7 warnings in 1235.72s (0:20:35) ================== ``` snappi_tests/multidut/pfcwd/test_multidut_pfcwd_burst_storm_with_snappi.py ``` -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html - generated xml file: /var/src/sonic-mgmt_vmsvc5-t2-8800-ixia_669de75fbbfea63e8b73b319/tests/logs/snappi_tests/multidut/pfcwd/test_multidut_pfcwd_burst_storm_with_snappi.xml - ================== 2 passed, 5 warnings in 858.60s (0:14:18) =================== ``` snappi_tests/multidut/pfcwd/test_multidut_pfcwd_basic_with_snappi.py (known failure on T2, fail as expected) ``` SKIPPED [4] snappi_tests/multidut/pfcwd/test_multidut_pfcwd_basic_with_snappi.py:190: Reboot type fast is not supported on cisco-8000 switches FAILED snappi_tests/multidut/pfcwd/test_multidut_pfcwd_basic_with_snappi.py::test_pfcwd_basic_multi_lossless_prio_restart_service[multidut_port_info0-True-swss] ===== 1 failed, 23 passed, 16 skipped, 27 warnings in 27779.49s (7:42:59) ====== ``` snappi_tests/multidut/pfcwd/test_multidut_pfcwd_a2a_with_snappi.py ``` -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html - generated xml file: /var/src/sonic-mgmt_vmsvc5-t2-8800-ixia_669de75fbbfea63e8b73b319/tests/logs/snappi_tests/multidut/pfcwd/test_multidut_pfcwd_a2a_with_snappi.xml - ================== 4 passed, 7 warnings in 1157.40s (0:19:17) ================== ``` snappi_tests/multidut/pfcwd/test_multidut_pfcwd_runtime_traffic_with_snappi.py ``` -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html - generated xml file: /var/src/sonic-mgmt_vmsvc5-t2-8800-ixia_669de75fbbfea63e8b73b319/tests/logs/snappi_tests/multidut/pfcwd/test_multidut_pfcwd_runtime_traffic_with_snappi.xml - ================== 2 passed, 6 warnings in 1162.35s (0:19:22) ================== ```

Summary: Update the missing T0 T1 tests feature in multidut Fixes # (issue) 30929279 The following PR were migrated for pfcwd [Snappi] Increase number of calls for ARP wait sonic-net#7370 [Snappi] PFCWD Tolerance Fixes sonic-net#9567 [Snappi] Modifications to pfcwd a2a test design for mlnx 4600c sonic-net#7367 to multidut code [Snappi] Modified pfcwd tests to add warm-up traffic sonic-net#7712 Signed-off-by: Austin Pham <austinpham@microsoft.com>

Summary: Update the missing T0 T1 tests feature in multidut Fixes # (issue) 30929279 The following PR were migrated for pfcwd [Snappi] Increase number of calls for ARP wait #7370 [Snappi] PFCWD Tolerance Fixes #9567 [Snappi] Modifications to pfcwd a2a test design for mlnx 4600c #7367 to multidut code [Snappi] Modified pfcwd tests to add warm-up traffic #7712 Signed-off-by: Austin Pham <austinpham@microsoft.com>

Summary: Update the missing T0 T1 tests feature in multidut Fixes # (issue) 30929279 The following PR were migrated for pfcwd [Snappi] Increase number of calls for ARP wait sonic-net#7370 [Snappi] PFCWD Tolerance Fixes sonic-net#9567 [Snappi] Modifications to pfcwd a2a test design for mlnx 4600c sonic-net#7367 to multidut code [Snappi] Modified pfcwd tests to add warm-up traffic sonic-net#7712 Signed-off-by: Austin Pham <austinpham@microsoft.com>

added warm-up traffic to all pfcwd tests

9e817d0

developfast added Request for 201911 branch Request for 202012 branch Request for 202205 branch labels Mar 10, 2023

developfast requested a review from baiwei0427 March 10, 2023 01:34

developfast mentioned this pull request Mar 10, 2023

[Snappi] Disable skip PFC watchdog tests for switches with broadcom ASICs #7718

Merged

6 tasks

neethajohn removed the Request for 201911 branch label Mar 10, 2023

baiwei0427 requested changes Mar 12, 2023

View reviewed changes

baiwei0427 reviewed Mar 13, 2023

View reviewed changes

resolved PR comments

7b9eb2b

developfast requested a review from baiwei0427 March 14, 2023 07:28

baiwei0427 approved these changes Mar 14, 2023

View reviewed changes

developfast requested a review from neethajohn March 14, 2023 07:32

bingwang-ms approved these changes Mar 14, 2023

View reviewed changes

developfast merged commit 0f53991 into sonic-net:master Mar 14, 2023

wangxin pushed a commit that referenced this pull request Mar 15, 2023

[Snappi] Modified pfcwd tests to add warm-up traffic (#7712)

dc162a6

* added warm-up traffic to all pfcwd tests

wangxin added the Included in 202205 branch label Mar 15, 2023

wangxin pushed a commit that referenced this pull request Mar 15, 2023

[Snappi] Modified pfcwd tests to add warm-up traffic (#7712)

a8c4850

* added warm-up traffic to all pfcwd tests

wangxin added the Included in 202012 branch label Mar 15, 2023

auspham added a commit to auspham/sonic-mgmt that referenced this pull request Jan 21, 2025

fix: migrate sonic-net#7712

8bc7deb

Signed-off-by: Austin Pham <austinpham@microsoft.com>

auspham added a commit to auspham/sonic-mgmt that referenced this pull request Jan 21, 2025

fix: migrate sonic-net#7712 to pfcwd_multidut_runtime_traffic_helper

68f01f1

Signed-off-by: Austin Pham <austinpham@microsoft.com>

auspham mentioned this pull request Jan 24, 2025

[T0 T1 merge] update multidut pfcwd tests #16657

Merged

11 tasks

mssonicbld mentioned this pull request Jan 29, 2025

[action] [PR:16657] [T0 T1 merge] update multidut pfcwd tests Azure/sonic-mgmt.msft#29

Merged

11 tasks

mssonicbld mentioned this pull request Jan 29, 2025

[action] [PR:16657] [T0 T1 merge] update multidut pfcwd tests #16707

Merged

11 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Snappi] Modified pfcwd tests to add warm-up traffic#7712

[Snappi] Modified pfcwd tests to add warm-up traffic#7712
developfast merged 2 commits intosonic-net:masterfrom
developfast:dojha/warm_up_traffic_pfcwd

developfast commented Mar 10, 2023

Uh oh!

mssonicbld commented Mar 10, 2023

Uh oh!

neethajohn commented Mar 10, 2023

Uh oh!

baiwei0427 Mar 12, 2023

Uh oh!

baiwei0427 Mar 12, 2023

Uh oh!

developfast Mar 13, 2023

Uh oh!

baiwei0427 Mar 12, 2023

Uh oh!

baiwei0427 Mar 12, 2023

Uh oh!

developfast Mar 13, 2023

Uh oh!

baiwei0427 Mar 13, 2023

Uh oh!

developfast Mar 13, 2023 •

edited

Loading

Uh oh!

mssonicbld commented Mar 14, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Conversation

developfast commented Mar 10, 2023

Description of PR

Type of change

Back port request

Approach

What is the motivation for this PR?

How did you do it?

How did you verify/test it?

Any platform specific information?

Supported testbed topology if it's a new test case?

Documentation

Uh oh!

mssonicbld commented Mar 10, 2023

Uh oh!

neethajohn commented Mar 10, 2023

Uh oh!

baiwei0427 Mar 12, 2023

Choose a reason for hiding this comment

Uh oh!

baiwei0427 Mar 12, 2023

Choose a reason for hiding this comment

Uh oh!

developfast Mar 13, 2023

Choose a reason for hiding this comment

Uh oh!

baiwei0427 Mar 12, 2023

Choose a reason for hiding this comment

Uh oh!

baiwei0427 Mar 12, 2023

Choose a reason for hiding this comment

Uh oh!

developfast Mar 13, 2023

Choose a reason for hiding this comment

Uh oh!

baiwei0427 Mar 13, 2023

Choose a reason for hiding this comment

Uh oh!

developfast Mar 13, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mssonicbld commented Mar 14, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

developfast Mar 13, 2023 •

edited

Loading