Skip to content

Revert "enable pfcwd for backplane ports (#3759)"#3767

Merged
yejianquan merged 1 commit intosonic-net:masterfrom
sdszhang:backplane_pfcwd
Feb 17, 2025
Merged

Revert "enable pfcwd for backplane ports (#3759)"#3767
yejianquan merged 1 commit intosonic-net:masterfrom
sdszhang:backplane_pfcwd

Conversation

@sdszhang
Copy link
Contributor

@sdszhang sdszhang commented Feb 17, 2025

This reverts commit 2866ccd.

After PFCWD is enabled on backplane in #3759, qos/test_qos_sai.py::TestQosSai::testQosSaiPfcXoffLimit[single_dut_multi_asic-xoff_1] and all non single-asic cases failed with the following error:

======================================================================
ERROR: sai_qos_tests.PFCtest
----------------------------------------------------------------------
Traceback (most recent call last):
  File "saitests/py3/sai_qos_tests.py", line 1789, in runTest
    pkt, int(self.test_params['pg']), asic_type, pkts_num_egr_mem)
  File "saitests/py3/sai_qos_tests.py", line 582, in fill_leakout_plus_one
    asic_type, int(pkts_num_egr_mem))
  File "saitests/py3/sai_qos_tests.py", line 626, in fill_egress_plus_one
    pkts_num_egr_mem + max_packets, pg_cntrs_base[queue], pg_cntrs[queue]))
RuntimeError: fill_egress_plus_one: Failure, sent 7884 packets, SQ occupancy bytes rose from 0 to 0

----------------------------------------------------------------------

as egress port is tx_disabled, PFC is generated from egress port back to backplane and triggered pfcwd on supervisor card. hence resulting in packet drop.

admin@sonic-sup-1:~$ show pfcwd stat -d all
            QUEUE       STATUS    STORM DETECTED/RESTORED     TX OK/DROP    RX OK/DROP    TX LAST OK/DROP    RX LAST OK/DROP
-----------------  -----------  -------------------------  -------------  ------------  -----------------  -----------------
 Ethernet-BP448:3  operational                      11/11  35404/1018833           0/0             0/8880                0/0
Ethernet-BP2240:3  operational                        2/2    3601/980808           0/0        3601/979813                0/0
Ethernet-BP2240:4  operational                        1/1          0/997           0/0              0/997                0/0
Ethernet-BP2244:3  operational                        2/2      7465/1990           0/0           2055/995                0/0
Ethernet-BP2244:4  operational                        1/1          1/994           0/0              1/994                0/0
Ethernet-BP4560:3  operational                        1/1        968/989           0/0            968/989                0/0
Ethernet-BP4560:4  operational                        1/1          1/983           0/0              1/983                0/0
Ethernet-BP4564:3  operational                        2/2        12/1974           0/0              7/988                0/0
Ethernet-BP4564:4  operational                        1/1          7/982           0/0              7/982                0/0
admin@sonic-sup-1:~$

What I did

Reverted back PR #3759

How I did it

Reverted back PR #3759

How to verify it

PFCtest passed after applying the patch manually. Full test_qos_sai is still running. will upload the result once it is done.

(Pdb) c
fill_egress_plus_one: Success, sent 6941 packets, SQ occupancy bytes rose from 0 to 1536
after send packets short of triggering PFC:
        recv_counters [0, 48251, 0, 0, 0, 1166406, 494656, 0, 0, 0, 21711040893, 1345, 48251, 0, 3019568, 1981, 0, 0]
        recv_counters_base [0, 48251, 0, 0, 0, 1166406, 494656, 0, 0, 0, 21700549309, 1344, 48251, 0, 3009234, 1980, 0, 0]
        xmit_counters [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 21763613747, 40252, 0, 0, 2352, 998, 0, 0]
        xmit_counters_base [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 21752352307, 40245, 0, 0, 2351, 998, 0, 0]

after send a few packets to trigger PFC:
        recv_counters [0, 48251, 0, 0, 0, 1238602, 494656, 0, 0, 0, 21717895805, 1345, 48251, 0, 3019609, 1981, 0, 0]
        recv_counters_base [0, 48251, 0, 0, 0, 1166406, 494656, 0, 0, 0, 21711040893, 1345, 48251, 0, 3019568, 1981, 0, 0]
        xmit_counters [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 21769594355, 40252, 0, 0, 2352, 999, 0, 0]
        xmit_counters_base [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 21752352307, 40245, 0, 0, 2351, 998, 0, 0]

after send packets short of ingress drop:
        recv_counters [0, 48251, 0, 0, 0, 1359193, 494656, 0, 0, 0, 21723777533, 1345, 48251, 0, 3019857, 1982, 0, 0]
        recv_counters_base [0, 48251, 0, 0, 0, 1238602, 494656, 0, 0, 0, 21717895805, 1345, 48251, 0, 3019609, 1981, 0, 0]
        xmit_counters [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 21776005939, 40252, 0, 0, 2353, 999, 0, 0]
        xmit_counters_base [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 21752352307, 40245, 0, 0, 2351, 998, 0, 0]

after send a few packets to trigger drop:
        recv_counters [0, 48272, 0, 0, 0, 1459846, 494656, 0, 0, 0, 21729769085, 1346, 48272, 0, 3019898, 1982, 0, 0]
        recv_counters_base [0, 48251, 0, 0, 0, 1359193, 494656, 0, 0, 0, 21723777533, 1345, 48251, 0, 3019857, 1982, 0, 0]
        xmit_counters [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 21781582707, 40252, 0, 0, 2354, 999, 0, 0]
        xmit_counters_base [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 21752352307, 40245, 0, 0, 2351, 998, 0, 0]

ok

----------------------------------------------------------------------
Ran 1 test in 65.702s

OK

Previous command output (if the output of a command-line utility has changed)

New command output (if the output of a command-line utility has changed)

@mssonicbld
Copy link
Collaborator

/azp run

@azure-pipelines
Copy link

Azure Pipelines successfully started running 1 pipeline(s).

@sdszhang
Copy link
Contributor Author

Copy link
Contributor

@yejianquan yejianquan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@mssonicbld
Copy link
Collaborator

Cherry-pick PR to msft-202405: Azure/sonic-utilities.msft#122

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants