[crm] Fix test failures on SN4700 by batching neighbor creation#19003
Merged
bingwang-ms merged 3 commits intosonic-net:masterfrom Aug 6, 2025
Merged
[crm] Fix test failures on SN4700 by batching neighbor creation#19003bingwang-ms merged 3 commits intosonic-net:masterfrom
bingwang-ms merged 3 commits intosonic-net:masterfrom
Conversation
test_crm_nexthop_group was failing on sn4700 testbeds due to a known orchagent crash. The crash is caused by a few factors: - Expected neighbor entry missing from kernel, causing tunnel-route to be programmed for unresolved nexthop. - tunnel-route being removed when nexthop group was deleted - route bulker ending early due to ITEM_NOT_FOUND (not processing remaining bulk removes) - next-hop group delete failing due to OBJECT_IN_USE. This PR addresses the first issue. During neighbor generation for the test, some of the neighbor updates were being missed from APPL_DB on sn4700 devices. Previously, the neighbors were programmed in the kernel in a single ip command, this fix splits up the neighbor programming into batches in order to prevent updates from being dropped. Signed-off-by: Nikola Dancejic <[email protected]>
Collaborator
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
Collaborator
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
Collaborator
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
Collaborator
|
@Ndancejic Can you help me understand |
Contributor
Author
I noticed that some of the neighbor entries being added at the start of the test were not being notified in appdb. appdb gets these from netlink when programming neighbors through the kernel. From what I understand if there's a high volume, these can be dropped from the kernel sometimes. In this case we were programming ~2000 entries in a single |
bingwang-ms
approved these changes
Aug 6, 2025
mssonicbld
pushed a commit
to mssonicbld/sonic-mgmt
that referenced
this pull request
Aug 6, 2025
…c-net#19003) * [crm] Fix test failures on SN4700 by batching neighbor creation test_crm_nexthop_group was failing on sn4700 testbeds due to a known orchagent crash. The crash is caused by a few factors: - Expected neighbor entry missing from kernel, causing tunnel-route to be programmed for unresolved nexthop. - tunnel-route being removed when nexthop group was deleted - route bulker ending early due to ITEM_NOT_FOUND (not processing remaining bulk removes) - next-hop group delete failing due to OBJECT_IN_USE. This PR addresses the first issue. During neighbor generation for the test, some of the neighbor updates were being missed from APPL_DB on sn4700 devices. Previously, the neighbors were programmed in the kernel in a single ip command, this fix splits up the neighbor programming into batches in order to prevent updates from being dropped. Signed-off-by: Nikola Dancejic <[email protected]> * fix pre-commit errors * fix pre-commit --------- Signed-off-by: Nikola Dancejic <[email protected]>
Collaborator
|
Cherry-pick PR to 202411: #20092 |
11 tasks
nissampa
pushed a commit
to nissampa/sonic-mgmt_dpu_test
that referenced
this pull request
Aug 7, 2025
…c-net#19003) * [crm] Fix test failures on SN4700 by batching neighbor creation test_crm_nexthop_group was failing on sn4700 testbeds due to a known orchagent crash. The crash is caused by a few factors: - Expected neighbor entry missing from kernel, causing tunnel-route to be programmed for unresolved nexthop. - tunnel-route being removed when nexthop group was deleted - route bulker ending early due to ITEM_NOT_FOUND (not processing remaining bulk removes) - next-hop group delete failing due to OBJECT_IN_USE. This PR addresses the first issue. During neighbor generation for the test, some of the neighbor updates were being missed from APPL_DB on sn4700 devices. Previously, the neighbors were programmed in the kernel in a single ip command, this fix splits up the neighbor programming into batches in order to prevent updates from being dropped. Signed-off-by: Nikola Dancejic <[email protected]> * fix pre-commit errors * fix pre-commit --------- Signed-off-by: Nikola Dancejic <[email protected]>
mssonicbld
pushed a commit
to mssonicbld/sonic-mgmt
that referenced
this pull request
Aug 8, 2025
…c-net#19003) * [crm] Fix test failures on SN4700 by batching neighbor creation test_crm_nexthop_group was failing on sn4700 testbeds due to a known orchagent crash. The crash is caused by a few factors: - Expected neighbor entry missing from kernel, causing tunnel-route to be programmed for unresolved nexthop. - tunnel-route being removed when nexthop group was deleted - route bulker ending early due to ITEM_NOT_FOUND (not processing remaining bulk removes) - next-hop group delete failing due to OBJECT_IN_USE. This PR addresses the first issue. During neighbor generation for the test, some of the neighbor updates were being missed from APPL_DB on sn4700 devices. Previously, the neighbors were programmed in the kernel in a single ip command, this fix splits up the neighbor programming into batches in order to prevent updates from being dropped. Signed-off-by: Nikola Dancejic <[email protected]> * fix pre-commit errors * fix pre-commit --------- Signed-off-by: Nikola Dancejic <[email protected]>
Collaborator
|
Cherry-pick PR to 202505: #20138 |
11 tasks
mssonicbld
pushed a commit
that referenced
this pull request
Aug 8, 2025
* [crm] Fix test failures on SN4700 by batching neighbor creation test_crm_nexthop_group was failing on sn4700 testbeds due to a known orchagent crash. The crash is caused by a few factors: - Expected neighbor entry missing from kernel, causing tunnel-route to be programmed for unresolved nexthop. - tunnel-route being removed when nexthop group was deleted - route bulker ending early due to ITEM_NOT_FOUND (not processing remaining bulk removes) - next-hop group delete failing due to OBJECT_IN_USE. This PR addresses the first issue. During neighbor generation for the test, some of the neighbor updates were being missed from APPL_DB on sn4700 devices. Previously, the neighbors were programmed in the kernel in a single ip command, this fix splits up the neighbor programming into batches in order to prevent updates from being dropped. Signed-off-by: Nikola Dancejic <[email protected]> * fix pre-commit errors * fix pre-commit --------- Signed-off-by: Nikola Dancejic <[email protected]>
mssonicbld
added
Included in 202505 branch
and removed
Created PR to 202505 branch
labels
ashutosh-agrawal
pushed a commit
to ashutosh-agrawal/sonic-mgmt
that referenced
this pull request
Aug 14, 2025
…c-net#19003) * [crm] Fix test failures on SN4700 by batching neighbor creation test_crm_nexthop_group was failing on sn4700 testbeds due to a known orchagent crash. The crash is caused by a few factors: - Expected neighbor entry missing from kernel, causing tunnel-route to be programmed for unresolved nexthop. - tunnel-route being removed when nexthop group was deleted - route bulker ending early due to ITEM_NOT_FOUND (not processing remaining bulk removes) - next-hop group delete failing due to OBJECT_IN_USE. This PR addresses the first issue. During neighbor generation for the test, some of the neighbor updates were being missed from APPL_DB on sn4700 devices. Previously, the neighbors were programmed in the kernel in a single ip command, this fix splits up the neighbor programming into batches in order to prevent updates from being dropped. Signed-off-by: Nikola Dancejic <[email protected]> * fix pre-commit errors * fix pre-commit --------- Signed-off-by: Nikola Dancejic <[email protected]>
bingwang-ms
pushed a commit
that referenced
this pull request
Aug 19, 2025
…) (#20092) * [crm] Fix test failures on SN4700 by batching neighbor creation test_crm_nexthop_group was failing on sn4700 testbeds due to a known orchagent crash. The crash is caused by a few factors: - Expected neighbor entry missing from kernel, causing tunnel-route to be programmed for unresolved nexthop. - tunnel-route being removed when nexthop group was deleted - route bulker ending early due to ITEM_NOT_FOUND (not processing remaining bulk removes) - next-hop group delete failing due to OBJECT_IN_USE. This PR addresses the first issue. During neighbor generation for the test, some of the neighbor updates were being missed from APPL_DB on sn4700 devices. Previously, the neighbors were programmed in the kernel in a single ip command, this fix splits up the neighbor programming into batches in order to prevent updates from being dropped. * fix pre-commit errors * fix pre-commit --------- Signed-off-by: Nikola Dancejic <[email protected]> Co-authored-by: Nikola Dancejic <[email protected]>
mssonicbld
added
Included in 202411 branch
and removed
Created PR to 202411 branch
labels
vidyac86
pushed a commit
to vidyac86/sonic-mgmt
that referenced
this pull request
Oct 23, 2025
…c-net#19003) * [crm] Fix test failures on SN4700 by batching neighbor creation test_crm_nexthop_group was failing on sn4700 testbeds due to a known orchagent crash. The crash is caused by a few factors: - Expected neighbor entry missing from kernel, causing tunnel-route to be programmed for unresolved nexthop. - tunnel-route being removed when nexthop group was deleted - route bulker ending early due to ITEM_NOT_FOUND (not processing remaining bulk removes) - next-hop group delete failing due to OBJECT_IN_USE. This PR addresses the first issue. During neighbor generation for the test, some of the neighbor updates were being missed from APPL_DB on sn4700 devices. Previously, the neighbors were programmed in the kernel in a single ip command, this fix splits up the neighbor programming into batches in order to prevent updates from being dropped. Signed-off-by: Nikola Dancejic <[email protected]> * fix pre-commit errors * fix pre-commit --------- Signed-off-by: Nikola Dancejic <[email protected]>
opcoder0
pushed a commit
to opcoder0/sonic-mgmt
that referenced
this pull request
Dec 8, 2025
…c-net#19003) * [crm] Fix test failures on SN4700 by batching neighbor creation test_crm_nexthop_group was failing on sn4700 testbeds due to a known orchagent crash. The crash is caused by a few factors: - Expected neighbor entry missing from kernel, causing tunnel-route to be programmed for unresolved nexthop. - tunnel-route being removed when nexthop group was deleted - route bulker ending early due to ITEM_NOT_FOUND (not processing remaining bulk removes) - next-hop group delete failing due to OBJECT_IN_USE. This PR addresses the first issue. During neighbor generation for the test, some of the neighbor updates were being missed from APPL_DB on sn4700 devices. Previously, the neighbors were programmed in the kernel in a single ip command, this fix splits up the neighbor programming into batches in order to prevent updates from being dropped. Signed-off-by: Nikola Dancejic <[email protected]> * fix pre-commit errors * fix pre-commit --------- Signed-off-by: Nikola Dancejic <[email protected]>
gshemesh2
pushed a commit
to gshemesh2/sonic-mgmt
that referenced
this pull request
Dec 16, 2025
…c-net#19003) * [crm] Fix test failures on SN4700 by batching neighbor creation test_crm_nexthop_group was failing on sn4700 testbeds due to a known orchagent crash. The crash is caused by a few factors: - Expected neighbor entry missing from kernel, causing tunnel-route to be programmed for unresolved nexthop. - tunnel-route being removed when nexthop group was deleted - route bulker ending early due to ITEM_NOT_FOUND (not processing remaining bulk removes) - next-hop group delete failing due to OBJECT_IN_USE. This PR addresses the first issue. During neighbor generation for the test, some of the neighbor updates were being missed from APPL_DB on sn4700 devices. Previously, the neighbors were programmed in the kernel in a single ip command, this fix splits up the neighbor programming into batches in order to prevent updates from being dropped. Signed-off-by: Nikola Dancejic <[email protected]> * fix pre-commit errors * fix pre-commit --------- Signed-off-by: Nikola Dancejic <[email protected]> Signed-off-by: Guy Shemesh <[email protected]>
AharonMalkin
pushed a commit
to AharonMalkin/sonic-mgmt
that referenced
this pull request
Dec 16, 2025
…c-net#19003) * [crm] Fix test failures on SN4700 by batching neighbor creation test_crm_nexthop_group was failing on sn4700 testbeds due to a known orchagent crash. The crash is caused by a few factors: - Expected neighbor entry missing from kernel, causing tunnel-route to be programmed for unresolved nexthop. - tunnel-route being removed when nexthop group was deleted - route bulker ending early due to ITEM_NOT_FOUND (not processing remaining bulk removes) - next-hop group delete failing due to OBJECT_IN_USE. This PR addresses the first issue. During neighbor generation for the test, some of the neighbor updates were being missed from APPL_DB on sn4700 devices. Previously, the neighbors were programmed in the kernel in a single ip command, this fix splits up the neighbor programming into batches in order to prevent updates from being dropped. Signed-off-by: Nikola Dancejic <[email protected]> * fix pre-commit errors * fix pre-commit --------- Signed-off-by: Nikola Dancejic <[email protected]> Signed-off-by: Aharon Malkin <[email protected]>
gshemesh2
pushed a commit
to gshemesh2/sonic-mgmt
that referenced
this pull request
Dec 21, 2025
…c-net#19003) * [crm] Fix test failures on SN4700 by batching neighbor creation test_crm_nexthop_group was failing on sn4700 testbeds due to a known orchagent crash. The crash is caused by a few factors: - Expected neighbor entry missing from kernel, causing tunnel-route to be programmed for unresolved nexthop. - tunnel-route being removed when nexthop group was deleted - route bulker ending early due to ITEM_NOT_FOUND (not processing remaining bulk removes) - next-hop group delete failing due to OBJECT_IN_USE. This PR addresses the first issue. During neighbor generation for the test, some of the neighbor updates were being missed from APPL_DB on sn4700 devices. Previously, the neighbors were programmed in the kernel in a single ip command, this fix splits up the neighbor programming into batches in order to prevent updates from being dropped. Signed-off-by: Nikola Dancejic <[email protected]> * fix pre-commit errors * fix pre-commit --------- Signed-off-by: Nikola Dancejic <[email protected]> Signed-off-by: Guy Shemesh <[email protected]>
venu-nexthop
pushed a commit
to venu-nexthop/sonic-mgmt
that referenced
this pull request
Jan 13, 2026
…c-net#19003) * [crm] Fix test failures on SN4700 by batching neighbor creation test_crm_nexthop_group was failing on sn4700 testbeds due to a known orchagent crash. The crash is caused by a few factors: - Expected neighbor entry missing from kernel, causing tunnel-route to be programmed for unresolved nexthop. - tunnel-route being removed when nexthop group was deleted - route bulker ending early due to ITEM_NOT_FOUND (not processing remaining bulk removes) - next-hop group delete failing due to OBJECT_IN_USE. This PR addresses the first issue. During neighbor generation for the test, some of the neighbor updates were being missed from APPL_DB on sn4700 devices. Previously, the neighbors were programmed in the kernel in a single ip command, this fix splits up the neighbor programming into batches in order to prevent updates from being dropped. Signed-off-by: Nikola Dancejic <[email protected]> * fix pre-commit errors * fix pre-commit --------- Signed-off-by: Nikola Dancejic <[email protected]>
gshemesh2
pushed a commit
to gshemesh2/sonic-mgmt
that referenced
this pull request
Jan 26, 2026
…c-net#19003) * [crm] Fix test failures on SN4700 by batching neighbor creation test_crm_nexthop_group was failing on sn4700 testbeds due to a known orchagent crash. The crash is caused by a few factors: - Expected neighbor entry missing from kernel, causing tunnel-route to be programmed for unresolved nexthop. - tunnel-route being removed when nexthop group was deleted - route bulker ending early due to ITEM_NOT_FOUND (not processing remaining bulk removes) - next-hop group delete failing due to OBJECT_IN_USE. This PR addresses the first issue. During neighbor generation for the test, some of the neighbor updates were being missed from APPL_DB on sn4700 devices. Previously, the neighbors were programmed in the kernel in a single ip command, this fix splits up the neighbor programming into batches in order to prevent updates from being dropped. Signed-off-by: Nikola Dancejic <[email protected]> * fix pre-commit errors * fix pre-commit --------- Signed-off-by: Nikola Dancejic <[email protected]> Signed-off-by: Guy Shemesh <[email protected]>
ytzur1
pushed a commit
to ytzur1/sonic-mgmt
that referenced
this pull request
Feb 2, 2026
…c-net#19003) * [crm] Fix test failures on SN4700 by batching neighbor creation test_crm_nexthop_group was failing on sn4700 testbeds due to a known orchagent crash. The crash is caused by a few factors: - Expected neighbor entry missing from kernel, causing tunnel-route to be programmed for unresolved nexthop. - tunnel-route being removed when nexthop group was deleted - route bulker ending early due to ITEM_NOT_FOUND (not processing remaining bulk removes) - next-hop group delete failing due to OBJECT_IN_USE. This PR addresses the first issue. During neighbor generation for the test, some of the neighbor updates were being missed from APPL_DB on sn4700 devices. Previously, the neighbors were programmed in the kernel in a single ip command, this fix splits up the neighbor programming into batches in order to prevent updates from being dropped. Signed-off-by: Nikola Dancejic <[email protected]> * fix pre-commit errors * fix pre-commit --------- Signed-off-by: Nikola Dancejic <[email protected]> Signed-off-by: Yael Tzur <[email protected]>
venu-nexthop
pushed a commit
to venu-nexthop/sonic-mgmt
that referenced
this pull request
Mar 27, 2026
…c-net#19003) * [crm] Fix test failures on SN4700 by batching neighbor creation test_crm_nexthop_group was failing on sn4700 testbeds due to a known orchagent crash. The crash is caused by a few factors: - Expected neighbor entry missing from kernel, causing tunnel-route to be programmed for unresolved nexthop. - tunnel-route being removed when nexthop group was deleted - route bulker ending early due to ITEM_NOT_FOUND (not processing remaining bulk removes) - next-hop group delete failing due to OBJECT_IN_USE. This PR addresses the first issue. During neighbor generation for the test, some of the neighbor updates were being missed from APPL_DB on sn4700 devices. Previously, the neighbors were programmed in the kernel in a single ip command, this fix splits up the neighbor programming into batches in order to prevent updates from being dropped. Signed-off-by: Nikola Dancejic <[email protected]> * fix pre-commit errors * fix pre-commit --------- Signed-off-by: Nikola Dancejic <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description of PR
test_crm_nexthop_group was failing on sn4700 testbeds due to a known orchagent crash. The crash is caused by a few factors:
This PR addresses the first issue.
During neighbor generation for the test, some of the neighbor updates were being missed from APPL_DB on sn4700 devices. Previously, the neighbors were programmed in the kernel in a single ip command, this fix splits up the neighbor programming into batches in order to prevent updates from being dropped.
Summary:
Fixes sonic-net/sonic-buildimage#21243
Type of change
Back port request
Approach
What is the motivation for this PR?
CRM test was failing on dualtor sn4700 platforms
How did you do it?
buffered neighbor creation to prevent updates from being dropped
How did you verify/test it?
ran test case on sn4700 dualto testbed
Any platform specific information?
only changes test run on sn4700
Documentation
ado: #33294907