[portsorch] fix PortsOrch::allPortsReady() returns true when it should not by stepanblyschak · Pull Request #1103 · sonic-net/sonic-swss

stepanblyschak · 2019-10-23T14:51:16Z

Warm start flow before the change:

1st iteration:

- BufferOrch::doTask(): returns since PortInitDone hasn't arrived yet
- PortsOrch::doTask():  processes all PORT_TABLE untill PortInitDone flag
                        m_pendingPortSet is empty yet and m_portInitDone is true
                        so allPortsReady() will return true
- AnyOrch::doTask():    check g_portsOrch->allPortsReady()

2nd iteration:

- BufferOrch::doTask(): now buffers are applied

This causes BufferOrch override PfcWdOrch's zero-buffer profile.

The change swaps BufferOrch and PortsOrch in m_orchList, because 1st
BufferOrch iteration will always skip processing and eliminates possibility
of having m_pendingPortSet not filled with ports after m_initDone is set to true.

Signed-off-by: Stepan Blyschak [email protected]

What I did

Why I did it

How I verified it

Details if related

orchagent/portsorch.cpp

wendani · 2019-10-25T02:00:51Z

orchagent/orchdaemon.cpp

     * That is ensured implicitly by the order of map key, "LAG_TABLE" is smaller than "VLAN_TABLE" in lexicographic order.
     */
-    m_orchList = { gSwitchOrch, gCrmOrch, gBufferOrch, gPortsOrch, gIntfsOrch, gNeighOrch, gRouteOrch, copp_orch, tunnel_decap_orch, qos_orch, wm_orch, policer_orch };
+    m_orchList = { gSwitchOrch, gCrmOrch, gPortsOrch, gBufferOrch, gIntfsOrch, gNeighOrch, gRouteOrch, copp_orch, tunnel_decap_orch, qos_orch, wm_orch, policer_orch };


This is a better order as BufferOrch::doTask() needs to wait for PortsOrch::isInitDone() to be true to proceed

It is interesting that BufferOrch was placed behind PortsPorch earlier as of this PR #515

orchagent/pfcwdorch.cpp

orchagent/portsorch.cpp

stepanblyschak · 2019-10-25T08:15:46Z

retest this please

stepanblyschak · 2019-10-25T11:08:47Z

Retest this please

qiluo-msft · 2019-10-26T00:44:19Z

orchagent/orchdaemon.cpp

     * That is ensured implicitly by the order of map key, "LAG_TABLE" is smaller than "VLAN_TABLE" in lexicographic order.
     */
-    m_orchList = { gSwitchOrch, gCrmOrch, gBufferOrch, gPortsOrch, gIntfsOrch, gNeighOrch, gRouteOrch, copp_orch, tunnel_decap_orch, qos_orch, wm_orch, policer_orch };
+    m_orchList = { gSwitchOrch, gCrmOrch, gPortsOrch, gBufferOrch, gIntfsOrch, gNeighOrch, gRouteOrch, copp_orch, tunnel_decap_orch, qos_orch, wm_orch, policer_orch };


gPortsOrch [](start = 42, length = 10)

Do not reorder if you don't have a strong reason. Any order should work for warm-reboot if we iterate enough rounds. #Closed

Agree with Qi

@qiluo-msft @prsunny I agree that it is just a matter of enough iterations, however this order makes more sense at least to me, so it's more a cosmetic change.
Besides, now I think with this order 3 instead of 4 iterations are needed in warm boot, let me check.
Do you have concern that this may break something? #Closed

Let me know if you have proof that 4 iterations -> 3. Otherwise I don't see any improvement.

In reply to: 339371158 [](ancestors = 339371158)

Please check one more commit to have 3 iterations.

In any case, let me separate this change in another PR

qiluo-msft · 2019-10-26T01:01:03Z

return true;

Can we initialize m_pendingPortSet here to fix everything you are trying to solve? #Closed

Refers to: orchagent/portsorch.cpp:1469 in c6b9c08. [](commit_id = c6b9c08, deletion_comment = False)

qiluo-msft · 2019-10-26T01:02:38Z

This PR will be an import bug fix. To make lives easier, could you please provide a vs test case, which failed currently version but fixed by this PR? #Closed

qiluo-msft

Commented

wendani · 2019-10-26T01:10:48Z

Stepan, as you analyze in sonic-net/sonic-mgmt#834 (comment), design of m_pendingPortSet has an issue. If we can populate all physical ports to m_pendingPortSet in the PortsOrch::bake() phase, we can drop the change to move the section of code down pasted below:

1681             if (!m_portConfigDone)
1682             {
1683                 // Not yet receive PortConfigDone. Save it for future retry
1684                 it++;
1685                 continue;
1686             }

#1103 (comment)

stepanblyschak · 2019-10-27T23:14:25Z

@qiluo-msft @wendani Not everything in this PR can be fixed by initializing m_pendingPortSet in bake().
Originaly I assumed that under some circumstances in cold boot there could be 'PortConfigDone' and 'PortInitDone' at the same time in m_toSync. Now, it looks like it is imposible to achive in cold boot.
So, yes, to make the fix more explicit I did a change to populate m_pendingPortSet in PortsOrch::bake().

Another issue is with Pfc wd start action is not protected by allPortsReady() causing errors in logs in warm boot and it is fixed as part of this PR, not related to m_pendingPortSet.

stepanblyschak · 2019-10-27T23:37:20Z

@qiluo-msft
Regarding VS test case, usually for such change it is hard to create VS test that checks the initialization order.
I could create a VS for pfcwd zero-buffer approach and, before warm restart, simulate storm using redis channel, but I do not see any pfcwd related tests in swss and don't think it will somehow work on VS.
If you have ideas I can try.

stepanblyschak · 2019-10-28T00:02:39Z

retest this please

qiluo-msft · 2019-10-28T05:39:29Z

Regarding test, I don't mean an end-to-end simulation of a warm-reboot plus PFC storm. Maybe an google test (unit test) could help here. You could just prepare some mock data in Redis or mock redis, and let orchagent consume them.

This test case should fail old code, but pass your new code.

In reply to: 546554272 [](ancestors = 546554272)

stepanblyschak · 2019-10-28T15:35:39Z

@qiluo-msft google test (unit test) would be simpler to create than VS, but it fails to compile tests in tests/mock_tests on recent master and I don't see it is run on PR. Are these tests expected to work? #Resolved

qiluo-msft · 2019-10-28T17:43:55Z

I recently fixed the google test. If you check vs test result you will find it included.

PASS: tests
============================================================================
Testsuite summary for sonic-swss 1.0
============================================================================
# TOTAL: 1
# PASS:  1
# SKIP:  0
# XFAIL: 0
# FAIL:  0
# XPASS: 0
# ERROR: 0
============================================================================

tests/mock_tests are not currently included

In reply to: 547004643 [](ancestors = 547004643)

…d not Warm start flow before the change: 1st iteration: - BufferOrch::doTask(): returns since PortInitDone hasn't arived yet - PortsOrch::doTask(): processes all PORT_TABLE untill PortInitDone flag m_pendingPortSet is empty yet and m_portInitDone is true so allPortsReady() will return true - AnyOrch::doTask(): check g_portsOrch->allPortsRead() 2nd iteration: - BufferOrch::doTask(): now buffers are applied This causes BufferOrch override PfcWdOrch's zero-buffer profile. The change swaps BufferOrch and PortsOrch in m_orchList, because 1st BufferOrch iteration will always skip processing and eliminates possibility of having m_pendingPortSet not filled with ports after m_initDone is set to true. Signed-off-by: Stepan Blyschak <[email protected]>

Signed-off-by: Stepan Blyschak <[email protected]>

… started in warm boot It appeared that pfc watchdog relied on a buggy behaviour of PortsOrch::allPortsReady(). In fixed PortsOrch::allPortsReady() you'll see that watchdog action is trying to start before watchdog was started, because allPortsReady() in PfcWdOrch::doTask() returned false. Before the fix watchdog was started before, because allPortsReady() lied that ports are ready when they were not. Signed-off-by: Stepan Blyschak <[email protected]>

Signed-off-by: Stepan Blyschak <[email protected]>

This reverts commit 84f80e0.

Signed-off-by: Stepan Blyschak <[email protected]>

stepanblyschak · 2019-10-31T10:31:59Z

Running main() from gtest_main.cc
[==========] Running 10 tests from 4 test cases.
[----------] Global test environment set-up.
[----------] 2 tests from PortsOrchTest
[ RUN      ] PortsOrchTest.PortReadinessColdBoot
[       OK ] PortsOrchTest.PortReadinessColdBoot (26 ms)
[ RUN      ] PortsOrchTest.PortReadinessWarmBoot
[       OK ] PortsOrchTest.PortReadinessWarmBoot (16 ms)
[----------] 2 tests from PortsOrchTest (42 ms total)

[----------] 1 test from AclTest
[ RUN      ] AclTest.Create_L3_Acl_Table
[       OK ] AclTest.Create_L3_Acl_Table (1 ms)
[----------] 1 test from AclTest (1 ms total)

[----------] 3 tests from AclOrchTest
[ RUN      ] AclOrchTest.ACL_Creation_and_Destorying
[       OK ] AclOrchTest.ACL_Creation_and_Destorying (1000 ms)
[ RUN      ] AclOrchTest.L3Acl_Matches_Actions
[       OK ] AclOrchTest.L3Acl_Matches_Actions (1001 ms)
[ RUN      ] AclOrchTest.L3V6Acl_Matches_Actions
[       OK ] AclOrchTest.L3V6Acl_Matches_Actions (1000 ms)
[----------] 3 tests from AclOrchTest (3001 ms total)

[----------] 4 tests from SaiSpy
[ RUN      ] SaiSpy.CURD
[       OK ] SaiSpy.CURD (0 ms)
[ RUN      ] SaiSpy.Same_Function_Signature_In_Same_API_Table
[       OK ] SaiSpy.Same_Function_Signature_In_Same_API_Table (1 ms)
[ RUN      ] SaiSpy.Same_Function_Signature_In_Different_API_Table
[       OK ] SaiSpy.Same_Function_Signature_In_Different_API_Table (0 ms)
[ RUN      ] SaiSpy.create_switch_and_acl_table
[       OK ] SaiSpy.create_switch_and_acl_table (0 ms)
[----------] 4 tests from SaiSpy (1 ms total)

[----------] Global test environment tear-down
[==========] 10 tests from 4 test cases ran. (4004 ms total)
[  PASSED  ] 10 tests.

Without fix:

Running main() from gtest_main.cc
[==========] Running 10 tests from 4 test cases.
[----------] Global test environment set-up.
[----------] 2 tests from PortsOrchTest
[ RUN      ] PortsOrchTest.PortReadinessColdBoot
[       OK ] PortsOrchTest.PortReadinessColdBoot (28 ms)
[ RUN      ] PortsOrchTest.PortReadinessWarmBoot
portsorch_ut.cpp:265: Failure
Value of: gPortsOrch->allPortsReady()
  Actual: true
Expected: false
[  FAILED  ] PortsOrchTest.PortReadinessWarmBoot (14 ms)
[----------] 2 tests from PortsOrchTest (42 ms total)

[----------] 1 test from AclTest
[ RUN      ] AclTest.Create_L3_Acl_Table
[       OK ] AclTest.Create_L3_Acl_Table (1 ms)
[----------] 1 test from AclTest (1 ms total)

[----------] 3 tests from AclOrchTest
[ RUN      ] AclOrchTest.ACL_Creation_and_Destorying
[       OK ] AclOrchTest.ACL_Creation_and_Destorying (1000 ms)
[ RUN      ] AclOrchTest.L3Acl_Matches_Actions
[       OK ] AclOrchTest.L3Acl_Matches_Actions (1001 ms)
[ RUN      ] AclOrchTest.L3V6Acl_Matches_Actions
[       OK ] AclOrchTest.L3V6Acl_Matches_Actions (1000 ms)
[----------] 3 tests from AclOrchTest (3002 ms total)

[----------] 4 tests from SaiSpy
[ RUN      ] SaiSpy.CURD
[       OK ] SaiSpy.CURD (0 ms)
[ RUN      ] SaiSpy.Same_Function_Signature_In_Same_API_Table
[       OK ] SaiSpy.Same_Function_Signature_In_Same_API_Table (0 ms)
[ RUN      ] SaiSpy.Same_Function_Signature_In_Different_API_Table
[       OK ] SaiSpy.Same_Function_Signature_In_Different_API_Table (0 ms)
[ RUN      ] SaiSpy.create_switch_and_acl_table
[       OK ] SaiSpy.create_switch_and_acl_table (0 ms)
[----------] 4 tests from SaiSpy (0 ms total)

[----------] Global test environment tear-down
[==========] 10 tests from 4 test cases ran. (4004 ms total)
[  PASSED  ] 9 tests.
[  FAILED  ] 1 test, listed below:
[  FAILED  ] PortsOrchTest.PortReadinessWarmBoot

 1 FAILED TEST

stepanblyschak · 2019-10-31T15:11:59Z

retest this please

Signed-off-by: Stepan Blyschak <[email protected]>

yxieca · 2019-11-01T23:37:47Z

@stepanblyschak This change cannot be cherry-picked cleanly into 201811 branch. Please create an PR for 201811 branch. Thanks!

…onic-net#1103) Signed-off-by: Danny Allen <[email protected]>

kcudnik · 2023-09-04T15:09:23Z

tests/mock_tests/Makefile.am

+                $(top_srcdir)/orchagent/orchdaemon.cpp \
+                $(top_srcdir)/orchagent/orch.cpp \
+                $(top_srcdir)/orchagent/notifications.cpp \
+                $(top_srcdir)/orchagent/routeorch.cpp \
+                $(top_srcdir)/orchagent/neighorch.cpp \
+                $(top_srcdir)/orchagent/intfsorch.cpp \
+                $(top_srcdir)/orchagent/portsorch.cpp \
+                $(top_srcdir)/orchagent/copporch.cpp \
+                $(top_srcdir)/orchagent/tunneldecaporch.cpp \
+                $(top_srcdir)/orchagent/qosorch.cpp \
+                $(top_srcdir)/orchagent/bufferorch.cpp \
+                $(top_srcdir)/orchagent/mirrororch.cpp \
+                $(top_srcdir)/orchagent/fdborch.cpp \
+                $(top_srcdir)/orchagent/aclorch.cpp \
+                $(top_srcdir)/orchagent/saihelper.cpp \
+                $(top_srcdir)/orchagent/switchorch.cpp \
+                $(top_srcdir)/orchagent/pfcwdorch.cpp \
+                $(top_srcdir)/orchagent/pfcactionhandler.cpp \
+                $(top_srcdir)/orchagent/policerorch.cpp \
+                $(top_srcdir)/orchagent/crmorch.cpp \
+                $(top_srcdir)/orchagent/request_parser.cpp \
+                $(top_srcdir)/orchagent/vrforch.cpp \
+                $(top_srcdir)/orchagent/countercheckorch.cpp \
+                $(top_srcdir)/orchagent/vxlanorch.cpp \
+                $(top_srcdir)/orchagent/vnetorch.cpp \
+                $(top_srcdir)/orchagent/dtelorch.cpp \
+                $(top_srcdir)/orchagent/flexcounterorch.cpp \
+                $(top_srcdir)/orchagent/watermarkorch.cpp \
+                $(top_srcdir)/orchagent/chassisorch.cpp \
+                $(top_srcdir)/orchagent/sfloworch.cpp


@stepanblyschak @Pterosaur @nazariig @lguohan please make orchagent as library *.la, it will be one file, orchagent is already compiled, and here in tests you are compiling it again, which is twice the same compilation, it extend compilation time twice !, now it takes 19 minutes to compile and using it as a lib it could take 10 min

…d not (sonic-net#1103) * [portsorch] fix PortsOrch::allPortsReady() returns true when it should not Warm start flow before the change: 1st iteration: - BufferOrch::doTask(): returns since PortInitDone hasn't arived yet - PortsOrch::doTask(): processes all PORT_TABLE untill PortInitDone flag m_pendingPortSet is empty yet and m_portInitDone is true so allPortsReady() will return true - AnyOrch::doTask(): check g_portsOrch->allPortsRead() 2nd iteration: - BufferOrch::doTask(): now buffers are applied This causes BufferOrch override PfcWdOrch's zero-buffer profile. The change swaps BufferOrch and PortsOrch in m_orchList, because 1st BufferOrch iteration will always skip processing and eliminates possibility of having m_pendingPortSet not filled with ports after m_initDone is set to true. * remove extra newline * [pfcwdorch] fix PfcWdSwOrch::doTask() starts WD action when WD wasn't started in warm boot It appeared that pfc watchdog relied on a buggy behaviour of PortsOrch::allPortsReady(). In fixed PortsOrch::allPortsReady() you'll see that watchdog action is trying to start before watchdog was started, because allPortsReady() in PfcWdOrch::doTask() returned false. Before the fix watchdog was started before, because allPortsReady() lied that ports are ready when they were not. * [portsorch] populate m_pendingPortSet in PortsOrch::bake() * [portsorch] optimize to 3 iterations instead of 4 * Revert "[portsorch] optimize to 3 iterations instead of 4" * revert change of order in m_orchList * [mock_tests] fix tests build * [mock_test] create unittest for PortsOrch::allPortsReady cold/warm flows * [orchdaemon] fix removed sfloworch * [mock_tests] make mock_tests run on "make check"

wendani requested review from lguohan and pavel-shirshov October 23, 2019 16:45

stcheng reviewed Oct 23, 2019

View reviewed changes

orchagent/portsorch.cpp Outdated Show resolved Hide resolved

lguohan assigned pavel-shirshov Oct 23, 2019

stepanblyschak marked this pull request as ready for review October 24, 2019 16:24

wendani changed the title ~~[portsorch] fix PortsOrch::allPortsReady() returns true when it shoul…~~ [portsorch] fix PortsOrch::allPortsReady() returns true when it should not Oct 25, 2019

wendani reviewed Oct 25, 2019

View reviewed changes

orchagent/pfcwdorch.cpp Show resolved Hide resolved

wendani reviewed Oct 25, 2019

View reviewed changes

orchagent/portsorch.cpp Outdated Show resolved Hide resolved

wendani requested review from prsunny and qiluo-msft October 26, 2019 00:29

qiluo-msft reviewed Oct 26, 2019

View reviewed changes

qiluo-msft requested changes Oct 26, 2019

View reviewed changes

wendani added the Request for 201811 Branch label Oct 26, 2019

stepanblyschak force-pushed the fix_buffer_dep branch from 82207af to 51e8707 Compare October 27, 2019 23:07

wendani approved these changes Oct 28, 2019

View reviewed changes

Stepan Blyschak added 3 commits October 31, 2019 12:18

remove extra newline

2be41b4

Signed-off-by: Stepan Blyschak <[email protected]>

Stepan Blyschak added 6 commits October 31, 2019 12:18

[portsorch] populate m_pendingPortSet in PortsOrch::bake()

59ad10b

Signed-off-by: Stepan Blyschak <[email protected]>

[portsorch] optimize to 3 iterations instead of 4

c3fd43c

Signed-off-by: Stepan Blyschak <[email protected]>

Revert "[portsorch] optimize to 3 iterations instead of 4"

50bc218

This reverts commit 84f80e0.

revert change of order in m_orchList

bb345ba

Signed-off-by: Stepan Blyschak <[email protected]>

[mock_tests] fix tests build

39c950b

Signed-off-by: Stepan Blyschak <[email protected]>

[mock_test] create unittest for PortsOrch::allPortsReady cold/warm flows

6a2073b

Signed-off-by: Stepan Blyschak <[email protected]>

stepanblyschak force-pushed the fix_buffer_dep branch from 8de2a78 to 6a2073b Compare October 31, 2019 10:18

[orchdaemon] fix removed sfloworch

51c230e

Signed-off-by: Stepan Blyschak <[email protected]>

qiluo-msft approved these changes Oct 31, 2019

View reviewed changes

[mock_tests] make mock_tests run on "make check"

86b45eb

Signed-off-by: Stepan Blyschak <[email protected]>

qiluo-msft merged commit bab7b93 into sonic-net:master Nov 1, 2019

yxieca removed the Request for 201811 Branch label Nov 1, 2019

daall mentioned this pull request Nov 8, 2019

swss build fails if SAI library installed #1122

Closed

EdenGri pushed a commit to EdenGri/sonic-swss that referenced this pull request Feb 28, 2022

[drop counters] Fix configuration for counters with lowercase names (s…

144bccb

…onic-net#1103) Signed-off-by: Danny Allen <[email protected]>

kcudnik reviewed Sep 4, 2023

View reviewed changes

Conversation

stepanblyschak commented Oct 23, 2019 • edited by wendani Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

1st iteration:

2nd iteration:

Uh oh!

Uh oh!

wendani Oct 25, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wendani Oct 25, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

stepanblyschak commented Oct 25, 2019

Uh oh!

stepanblyschak commented Oct 25, 2019

Uh oh!

qiluo-msft Oct 26, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

prsunny Oct 26, 2019

Choose a reason for hiding this comment

Uh oh!

stepanblyschak Oct 27, 2019 • edited by qiluo-msft Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

qiluo-msft Oct 28, 2019

Choose a reason for hiding this comment

Uh oh!

stepanblyschak Oct 28, 2019

Choose a reason for hiding this comment

Uh oh!

stepanblyschak Oct 28, 2019

Choose a reason for hiding this comment

Uh oh!

qiluo-msft commented Oct 26, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

qiluo-msft commented Oct 26, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

qiluo-msft left a comment

Choose a reason for hiding this comment

Uh oh!

wendani commented Oct 26, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

stepanblyschak commented Oct 27, 2019

Uh oh!

stepanblyschak commented Oct 27, 2019

Uh oh!

stepanblyschak commented Oct 28, 2019

Uh oh!

qiluo-msft commented Oct 28, 2019

Uh oh!

stepanblyschak commented Oct 28, 2019 • edited by qiluo-msft Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

qiluo-msft commented Oct 28, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

stepanblyschak commented Oct 31, 2019

Uh oh!

stepanblyschak commented Oct 31, 2019

Uh oh!

yxieca commented Nov 1, 2019

Uh oh!

kcudnik Sep 4, 2023

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

stepanblyschak commented Oct 23, 2019 •

edited by wendani

Loading

wendani Oct 25, 2019 •

edited

Loading

wendani Oct 25, 2019 •

edited

Loading

qiluo-msft Oct 26, 2019 •

edited

Loading

stepanblyschak Oct 27, 2019 •

edited by qiluo-msft

Loading

qiluo-msft commented Oct 26, 2019 •

edited

Loading

qiluo-msft commented Oct 26, 2019 •

edited

Loading

wendani commented Oct 26, 2019 •

edited

Loading

stepanblyschak commented Oct 28, 2019 •

edited by qiluo-msft

Loading

qiluo-msft commented Oct 28, 2019 •

edited

Loading