Update all port pfcwd storm test to make it more stable

[echuawu]

Description of PR

Update all port pfcwd storm test to make it more stable

1. Optimize background traffic
2. Fix arp resolve issue
3. Remove ptf leftovers added in pfcwd test

Summary:
Fixes # (issue)
All port pfcwd storm test not stable enough.

Type of change

• Bug fix
• Testbed and Framework(new/improvement)
• Test case(new/improvement)

Back port request

• 202012
• 202205
• 202305
• 202311
• 202405

Approach

What is the motivation for this PR?

Update all port pfcwd storm test is not stable engough.

How did you do it?

Optimize background traffic and arp resolvation.

How did you verify/test it?

Run it in internal regression.

Any platform specific information?

Supported testbed topology if it's a new test case?

Documentation

[liat-grozovik]

@yxieca @bingwang-ms could you please review or suggest someone to review?

[bingwang-ms]

@kperumalbfn Can you help review if you get a chance?

[yxieca]

@lipxu please help review.

[mssonicbld]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[lipxu]

https://elastictest.org/scheduler/testplan/67643e9eb7a7fea3d1f4ede2?leftSideViewMode=detail
Passed internal tests on Broadcom devices.

[mssonicbld]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[echuawu]

/azpw run

[mssonicbld]

/AzurePipelines run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[mssonicbld]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[echuawu]

/azpw run

[mssonicbld]

/AzurePipelines run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[echuawu]

/azpw run

[mssonicbld]

/AzurePipelines run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[mssonicbld]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[echuawu]

Hi @bingwang-ms , the 2 checker failure were due to testbed related issue, I had retriggered them many times, but still could not pass them. Do you know from whom could I get help?

1. DUTHOST_UNREACHABLE
2025-01-03T04:53:27.8357425Z Error type: DUTHOST_UNREACHABLE
2025-01-03T04:53:27.8358232Z Error message: DUT host unreachable for vms-kvm-dual-t0 on running test_pretest.py|||vms-kvm-dual-t0_377521
2025-01-03T04:53:27.8359427Z Operation failed with exception: Exception('Test plan id: 677764fe98ec838ee83d611c, status: FAILED, result: FAILED, Elapsed 602 seconds. Check https://elastictest.org/scheduler/testplan/677764fe98ec838ee83d611c for test plan status')

2. UPGRADE_IMAGE_FAILED
2025-01-03T04:43:14.9498342Z Error type: UPGRADE_IMAGE_FAILED
2025-01-03T04:43:14.9499389Z Error message: Prepare testbed failed: testbed_q_sonic-elastictest-prod-vmss-E8s-v3_261321 for UPGRADE_IMAGE_FAILED, testbed_id is: 67776755e396f7257aacb936, cmd is: Download image
2025-01-03T04:43:14.9500664Z Operation failed with exception: Exception('Test plan id: 677764fbb9fbfafb6a51c2e7, status: FAILED, result: FAILED, Elapsed 782 seconds. Check https://elastictest.org/scheduler/testplan/677764fbb9fbfafb6a51c2e7 for test plan status')

[echuawu]

@echuawu Can you please address the comment? Feel free to let me know if an offline discussion is needed.

Hi @bingwang-ms , the key concern of you is the 100000 background packets had been changed to 500. The backgroud traffic would be sent from ports one by one. If the packet number is too large, it would lead to the full loop could not be finished within 1 second, then some ports would be failed to be detected by pfcwd. Setting the packet number to a small value would make sure pfcwd could detect storm on all the ports.

Hi @echuawu , my concern is still not addressed. If the packet number is set to 500, then ptf would be able to finish the packet sending at a very short time. It's possible that PFC pause is not sent out from leaf fanout. How can you guarantee this?

The background traffic should be received within a detect interval, it does not need to be received at the same time when PFC pause receiving. 500 packets is a proper background traffic that it could make sure all the ports could receive the background traffic(due to the background traffic was send from ptf port one by one). If set a big value, such as 100000, the send duration would be longer than 500 packets, it may lead to some ports could not detect the background traffic within a detect interval. That's what we met before, and then update the number from 100000 to 500.

[lolyu]

Hi @bingwang-ms, could you please help check this PR?

[mssonicbld]

Cherry-pick PR to 202411: #19088

[mssonicbld]

Cherry-pick PR to 202505: #19113

[congh-nvidia]

Hi @bingwang-ms we also need this in 202411, could you please help cherry-pick?
@lolyu this will fix the key error issue.