cisco-8000: set 512bit time to a longer duration for cisco-8000 pfcwd tests using pfc_gen.py by rraghav-cisco · Pull Request #16159 · sonic-net/sonic-mgmt

rraghav-cisco · 2024-12-19T06:03:26Z

Description of PR

Summary:
Fixes the flakiness of pfc_gen in pfcwd scripts for cisco-8000. We use a new debug CLI script to force the DUT to wait longer in case of a miss in pfc packets from the fanout due to pfc_gen script. So even if the pfc_gen/fanout misses a couple of pfc frames to DUT, the dut would still not send out data packets.

Type of change

Bug fix
Testbed and Framework(new/improvement)
Test case(new/improvement)

Back port request

Approach

What is the motivation for this PR?

Flakiness of pfc-gen. Particularly with 400G links.

How did you do it?

We have added a new dshell based script that will force the DUT to wait before transmitting data in case of a miss in pfc pause frames.

How did you verify/test it?

Ran on our duts, with 100G and 400G.

Any platform specific information?

The new fix specific only to cisco-8000.

mssonicbld · 2024-12-19T06:03:28Z

/azp run

azure-pipelines · 2024-12-19T06:03:40Z

Azure Pipelines successfully started running 1 pipeline(s).

sdszhang · 2024-12-19T22:47:47Z

@abdosi can you help to take a look ?

mssonicbld · 2024-12-19T23:22:19Z

/azp run

azure-pipelines · 2024-12-19T23:22:30Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2025-01-06T18:56:39Z

/azp run

azure-pipelines · 2025-01-06T18:56:53Z

Azure Pipelines successfully started running 1 pipeline(s).

sdszhang · 2025-01-07T03:49:52Z

tests/pfcwd/conftest.py

+
+
+@pytest.fixture(scope="function", autouse=False)
+def set_pfc_time_cisco_8000(


pls limit this change to Cisco T2 chassis only for now.

sdszhang · 2025-01-07T03:50:07Z

tests/pfcwd/conftest.py

+            port)
+
+
+def set_pfc_timer_cisco_8000(duthost, asic_id, script, port):


pls limit this change to Cisco T2 chassis only for now.

mssonicbld · 2025-01-07T06:26:29Z

/azp run

azure-pipelines · 2025-01-07T06:26:41Z

Azure Pipelines successfully started running 1 pipeline(s).

sdszhang · 2025-01-08T22:19:00Z

@rraghav-cisco can you upload the test result.

mssonicbld · 2025-01-08T22:44:41Z

/azp run

azure-pipelines · 2025-01-08T22:44:53Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2025-01-08T22:51:12Z

/azp run

azure-pipelines · 2025-01-08T22:51:24Z

Azure Pipelines successfully started running 1 pipeline(s).

yejianquan · 2025-01-10T00:21:48Z

/azp run

azure-pipelines · 2025-01-10T00:22:00Z

Azure Pipelines successfully started running 1 pipeline(s).

yejianquan · 2025-01-10T00:25:19Z

/azp run

azure-pipelines · 2025-01-10T00:25:28Z

Azure Pipelines successfully started running 1 pipeline(s).

yejianquan

LGTM

Description of PR Summary: Fixes the flakiness of pfc_gen in pfcwd scripts for cisco-8000. We use a new debug CLI script to force the DUT to wait longer in case of a miss in pfc packets from the fanout due to pfc_gen script. So even if the pfc_gen/fanout misses a couple of pfc frames to DUT, the dut would still not send out data packets. Approach What is the motivation for this PR? Flakiness of pfc-gen. Particularly with 400G links. How did you do it? We have added a new dshell based script that will force the DUT to wait before transmitting data in case of a miss in pfc pause frames. How did you verify/test it? Ran on our duts, with 100G and 400G. Any platform specific information? The new fix specific only to cisco-8000. co-authorized by: jianquanye@microsoft.com

mssonicbld · 2025-01-13T00:03:21Z

Cherry-pick PR to 202411: #16468

mssonicbld · 2025-01-13T00:03:23Z

Cherry-pick PR to 202405: #16469

Description of PR Summary: Fixes the flakiness of pfc_gen in pfcwd scripts for cisco-8000. We use a new debug CLI script to force the DUT to wait longer in case of a miss in pfc packets from the fanout due to pfc_gen script. So even if the pfc_gen/fanout misses a couple of pfc frames to DUT, the dut would still not send out data packets. Approach What is the motivation for this PR? Flakiness of pfc-gen. Particularly with 400G links. How did you do it? We have added a new dshell based script that will force the DUT to wait before transmitting data in case of a miss in pfc pause frames. How did you verify/test it? Ran on our duts, with 100G and 400G. Any platform specific information? The new fix specific only to cisco-8000. co-authorized by: jianquanye@microsoft.com

auspham · 2025-01-23T04:18:34Z

tests/pfcwd/conftest.py

+    asic_arg = ""
+    if asic_id:
+        asic_arg = f"-n asic{asic_id}"
+    duthost.shell(f"show platform npu script {asic_arg} -s {script_name}")


@rraghav-cisco i ran into this problem:

> raise RunAnsibleModuleFail("run module {} failed".format(self.module_name), res) E tests.common.errors.RunAnsibleModuleFail: run module shell failed, Ansible Results => E failed = True E changed = True E rc = 2 E cmd = show platform npu script -s set_pfc_time.py E start = 2025-01-23 04:05:01.590011 E end = 2025-01-23 04:05:03.007243 E delta = 0:00:01.417232 E msg = non-zero return code E invocation = {'module_args': {'_raw_params': 'show platform npu script -s set_pfc_time.py', '_uses_shell': True, 'warn': False, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}} E _ansible_no_log = None E stdout = E stderr = E Usage: show platform npu script [OPTIONS] E Try "show platform npu script -h" for help. E E Error: Missing option "-n". Choose from: E asic0, E asic1, E asic2.

cc @sdszhang

@rraghav-cisco i ran into this problem:

> raise RunAnsibleModuleFail("run module {} failed".format(self.module_name), res) E tests.common.errors.RunAnsibleModuleFail: run module shell failed, Ansible Results => E failed = True E changed = True E rc = 2 E cmd = show platform npu script -s set_pfc_time.py E start = 2025-01-23 04:05:01.590011 E end = 2025-01-23 04:05:03.007243 E delta = 0:00:01.417232 E msg = non-zero return code E invocation = {'module_args': {'_raw_params': 'show platform npu script -s set_pfc_time.py', '_uses_shell': True, 'warn': False, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}} E _ansible_no_log = None E stdout = E stderr = E Usage: show platform npu script [OPTIONS] E Try "show platform npu script -h" for help. E E Error: Missing option "-n". Choose from: E asic0, E asic1, E asic2.

cc @sdszhang

@auspham , @sdszhang : I have raised #17858 for this issue.

Description of PR Summary: Fixes the flakiness of pfc_gen in pfcwd scripts for cisco-8000. We use a new debug CLI script to force the DUT to wait longer in case of a miss in pfc packets from the fanout due to pfc_gen script. So even if the pfc_gen/fanout misses a couple of pfc frames to DUT, the dut would still not send out data packets. Approach What is the motivation for this PR? Flakiness of pfc-gen. Particularly with 400G links. How did you do it? We have added a new dshell based script that will force the DUT to wait before transmitting data in case of a miss in pfc pause frames. How did you verify/test it? Ran on our duts, with 100G and 400G. Any platform specific information? The new fix specific only to cisco-8000. co-authorized by: jianquanye@microsoft.com

Add pfc-timer-set to 500mS for pfcwd tests.

1f333a9

Adding the new cisco dshell scripts for pfc-set-timers.

71920af

Add the new fixture to all_port_storm as well.

e781ca6

sdszhang reviewed Jan 7, 2025

View reviewed changes

Limit the new fixture only to cisco-8000 and T2.

431740d

Update the set_pfc_time script to include other platforms.

87d5a80

Update default_pfc_timer to include all platforms.

6865d7a

sdszhang added chassis Request for 202405 branch Request for 202411 branch labels Jan 11, 2025

sdszhang approved these changes Jan 12, 2025

View reviewed changes

yejianquan approved these changes Jan 12, 2025

View reviewed changes

yejianquan merged commit ab3d76b into sonic-net:master Jan 13, 2025

yejianquan added Approved for 202405 branch Approved for 202411 branch labels Jan 13, 2025

mssonicbld added the Created PR to 202411 branch label Jan 13, 2025

mssonicbld added the Created PR to 202405 branch label Jan 13, 2025

This was referenced Jan 13, 2025

[action] [PR:16159] Add pfc-timer-set to 500mS for pfcwd tests. #16468

Merged

[action] [PR:16159] Add pfc-timer-set to 500mS for pfcwd tests. #16469

Merged

mssonicbld added Included in 202405 branch and removed Created PR to 202405 branch labels Jan 13, 2025

mssonicbld added Included in 202411 branch and removed Created PR to 202411 branch labels Jan 15, 2025

auspham reviewed Jan 23, 2025

View reviewed changes

sdszhang mentioned this pull request Jan 28, 2025

pfcwd: Set fake-storm false for cisco platforms. #16695

Merged

11 tasks

rraghav-cisco changed the title ~~Add pfc-timer-set to 500mS for pfcwd tests.~~ cisco-8000: set 512bit time to a longer duration for cisco-8000 pfcwd tests using pfc_gen.py Jan 29, 2025



		@pytest.fixture(scope="function", autouse=False)
		def set_pfc_time_cisco_8000(

		port)


		def set_pfc_timer_cisco_8000(duthost, asic_id, script, port):

Conversation

rraghav-cisco commented Dec 19, 2024

Description of PR

Type of change

Back port request

Approach

What is the motivation for this PR?

How did you do it?

How did you verify/test it?

Any platform specific information?

Uh oh!

mssonicbld commented Dec 19, 2024

Uh oh!

azure-pipelines bot commented Dec 19, 2024

Uh oh!

sdszhang commented Dec 19, 2024

Uh oh!

mssonicbld commented Dec 19, 2024

Uh oh!

azure-pipelines bot commented Dec 19, 2024

Uh oh!

mssonicbld commented Jan 6, 2025

Uh oh!

azure-pipelines bot commented Jan 6, 2025

Uh oh!

sdszhang Jan 7, 2025

Choose a reason for hiding this comment

Uh oh!

sdszhang Jan 7, 2025

Choose a reason for hiding this comment

Uh oh!

mssonicbld commented Jan 7, 2025

Uh oh!

azure-pipelines bot commented Jan 7, 2025

Uh oh!

sdszhang commented Jan 8, 2025

Uh oh!

mssonicbld commented Jan 8, 2025

Uh oh!

azure-pipelines bot commented Jan 8, 2025

Uh oh!

mssonicbld commented Jan 8, 2025

Uh oh!

azure-pipelines bot commented Jan 8, 2025

Uh oh!

yejianquan commented Jan 10, 2025

Uh oh!

azure-pipelines bot commented Jan 10, 2025

Uh oh!

yejianquan commented Jan 10, 2025

Uh oh!

azure-pipelines bot commented Jan 10, 2025

Uh oh!

yejianquan left a comment

Choose a reason for hiding this comment

Uh oh!

mssonicbld commented Jan 13, 2025

Uh oh!

mssonicbld commented Jan 13, 2025

Uh oh!

auspham Jan 23, 2025

Choose a reason for hiding this comment

Uh oh!

rraghav-cisco Apr 4, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants