Stabilize PFC watchdog test

[bingwang-ms]

Description of PR

Summary:
This PR is to stabilize PFC watchdog test.
Changes include

1. Do not use multi-process to send PFC pause frames if there is only one interface to be paused
   This is because I didn't see improvement using multiple processes to send PFC pause frames when there is only 1 port to be paused. And the multi-processing caused test flakiness in test_pfcwd_multi_port
2. Check PFCWD stats after triggering PFC pause on leaf fanout
   Before this change, there was a hard-coded 5 seconds delay. But on some platform (e.g, Mellanox) more time is needed until PFC watchdog is triggered.

Type of change

• Bug fix
• Testbed and Framework(new/improvement)
• Test case(new/improvement)

Back port request

• 202012
• 202205
• 202305
• 202311
• 202405

Approach

What is the motivation for this PR?

This PR is to stabilize PFC watchdog test.

How did you do it?

1. Do not use multi-process to send PFC pause frames if there is only one interface to be paused
2. Check PFCWD stats after triggering PFC pause on leaf fanout

How did you verify/test it?

The change is verified on a Mellanox testbed. All test cases in pfcwd/test_pfcwd_function.py are consistently passing now.

collected 5 items                                                                                                                                                                                     

pfcwd/test_pfcwd_function.py::TestPfcwdFunc::test_pfcwd_actions[str-msn2700a1-0 ^H ^HPASSED                                                                                                        [ 20%]
pfcwd/test_pfcwd_function.py::TestPfcwdFunc::test_pfcwd_multi_port[str-msn2700a1-03]  ^H ^HPASSED                                                                                                     [ 40%]
pfcwd/test_pfcwd_function.py::TestPfcwdFunc::test_pfcwd_mmu_change[str-msn2700a1-03]  ^HPASSED                                                                                                     [ 60%]
pfcwd/test_pfcwd_function.py::TestPfcwdFunc::test_pfcwd_port_toggle[str-msn2700a1-03]  ^H ^HPASSED                                                                                                    [ 80%]
pfcwd/test_pfcwd_function.py::TestPfcwdFunc::test_pfcwd_no_traffic[str-msn2700a1-03] SKIPPED (This test is applicable only for cisco-8000)                                                      [100%]

Any platform specific information?

No.

Supported testbed topology if it's a new test case?

Not a new test case.

Documentation

[wsycqyz]

Let me verify the PR with cisco platform.

[yanmo96]

LGTM

[bingwang-ms]

Pushed another commit to limit the change to Mellanox only, and also run the test on multiple platforms.
On Cisco testbed.

collected 5 items                                                                                                                                                                                     

pfcwd/test_pfcwd_function.py::TestPfcwdFunc::test_pfcwd_actions[str3-8111-05]  ^H ^HPASSED                                                                                                            [ 20%]
pfcwd/test_pfcwd_function.py::TestPfcwdFunc::test_pfcwd_multi_port[str3-8111-05]  ^H ^HPASSED                                                                                                         [ 40%]
pfcwd/test_pfcwd_function.py::TestPfcwdFunc::test_pfcwd_mmu_change[str3-8111-05]    ^HPASSED                                                                                                         [ 60%]
pfcwd/test_pfcwd_function.py::TestPfcwdFunc::test_pfcwd_port_toggle[str3-8111-05]  ^H ^HPASSED                                                                                                        [ 80%]
pfcwd/test_pfcwd_function.py::TestPfcwdFunc::test_pfcwd_no_traffic[str3-8111-05] PASSED  

The change is also verified on a SN4700 testbed with Mellanox leaf fanout.

[bingwang-ms]

@nhe-NV Can you help review this change?

[mssonicbld]

@bingwang-ms PR conflicts with 202405 branch

[bingwang-ms]

Awaiting PR #14041 to be cherry-picked.