Graceful shutdown vpp to avoid core dump by yue-fred-gao · Pull Request #1714 · sonic-net/sonic-sairedis

yue-fred-gao · 2025-12-05T18:19:43Z

why

when we shutdown vpp syncd, such as config reload, syncd produces a core due to ungraceful shutdown. Here is the stack trace.

#0  0x00007f191a069eec in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007f191a01afb2 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#2  0x00007f191a005472 in abort () from /lib/x86_64-linux-gnu/libc.so.6
#3  0x00007f191a27e919 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#4  0x00007f191a289e1a in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#5  0x00007f191a289e85 in std::terminate() () from /lib/x86_64-linux-gnu/libstdc++.so.6
#6  0x000056339ef8f7de in std::__terminate () at /usr/include/x86_64-linux-gnu/c++/12/bits/c++config.h:312
#7  std::thread::~thread (this=<optimized out>, __in_chrg=<optimized out>) at /usr/include/c++/12/bits/std_thread.h:151
#8  std::_Destroy<std::thread> (__pointer=<optimized out>) at /usr/include/c++/12/bits/stl_construct.h:151
#9  std::allocator_traits<std::allocator<void> >::destroy<std::thread> (__p=<optimized out>) at /usr/include/c++/12/bits/alloc_traits.h:648
#10 std::_Sp_counted_ptr_inplace<std::thread, std::allocator<void>, (__gnu_cxx::_Lock_policy)2>::_M_dispose (this=<optimized out>) at /usr/include/c++/12/bits/shared_ptr_base.h:613
#11 0x00007f191a59bd23 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release (this=0x5633b57bfb40) at /usr/include/c++/12/bits/shared_ptr_base.h:346
#12 std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release (this=0x5633b57bfb40) at /usr/include/c++/12/bits/shared_ptr_base.h:317
#13 std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count (this=0x5633b57c27c8, __in_chrg=<optimized out>) at /usr/include/c++/12/bits/shared_ptr_base.h:1071
#14 std::__shared_ptr<std::thread, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr (this=0x5633b57c27c0, __in_chrg=<optimized out>) at /usr/include/c++/12/bits/shared_ptr_base.h:1524
#15 std::shared_ptr<std::thread>::~shared_ptr (this=0x5633b57c27c0, __in_chrg=<optimized out>) at /usr/include/c++/12/bits/shared_ptr.h:175
#16 saivs::SwitchVpp::~SwitchVpp (this=0x5633b57c20b0, __in_chrg=<optimized out>) at vpp/SwitchVpp.h:44
#17 0x000056339ef65507 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release (this=0x5633b57c20a0) at /usr/include/c++/12/bits/shared_ptr_base.h:346
#18 std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release (this=0x5633b57c20a0) at /usr/include/c++/12/bits/shared_ptr_base.h:317
#19 0x00007f191a588fe3 in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count (this=<synthetic pointer>, __in_chrg=<optimized out>) at /usr/include/c++/12/bits/shared_ptr_base.h:1071
#20 std::__shared_ptr<saivs::SwitchStateBase, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr (this=<synthetic pointer>, __in_chrg=<optimized out>) at /usr/include/c++/12/bits/shared_ptr_base.h:1524
#21 std::shared_ptr<saivs::SwitchStateBase>::~shared_ptr (this=<synthetic pointer>, __in_chrg=<optimized out>) at /usr/include/c++/12/bits/shared_ptr.h:175
#22 saivs::VirtualSwitchSaiInterface::remove (this=this@entry=0x5633b57a2eb0, switchId=switchId@entry=141733920768, objectType=objectType@entry=SAI_OBJECT_TYPE_SWITCH, serializedObjectId="oid:0x2100000000")
    at ./vslib/VirtualSwitchSaiInterface.cpp:793
#23 0x00007f191a5894cc in saivs::VirtualSwitchSaiInterface::remove (this=0x5633b57a2eb0, objectType=SAI_OBJECT_TYPE_SWITCH, objectId=<optimized out>) at ./vslib/VirtualSwitchSaiInterface.cpp:354
#24 0x00007f191a8d214e in saimeta::Meta::remove (this=0x5633b57ab300, object_type=SAI_OBJECT_TYPE_SWITCH, object_id=<optimized out>) at ./meta/Meta.cpp:264
#25 0x00007f191a5643a2 in saivs::Sai::remove (this=0x5633b5795f50, objectType=SAI_OBJECT_TYPE_SWITCH, objectId=141733920768) at ./vslib/Sai.cpp:355
#26 0x00007f191a55eea5 in stub_remove_switch (switch_id=141733920768) at ./vslib/sai_vs.cpp:36
#27 0x000056339ef920d3 in syncd::VendorSai::remove (this=0x5633b57a3140, objectType=SAI_OBJECT_TYPE_SWITCH, objectId=141733920768) at ./syncd/VendorSai.cpp:206
#28 0x000056339ef69fdc in syncd::Syncd::removeAllSwitches (this=this@entry=0x5633b579f850) at ./syncd/Syncd.cpp:5516
#29 0x000056339ef85dd2 in syncd::Syncd::run (this=this@entry=0x5633b579f850) at ./syncd/Syncd.cpp:5960
#30 0x000056339ef65074 in syncd_main (argc=argc@entry=7, argv=argv@entry=0x7ffcc53269b8) at ./syncd/syncd_main.cpp:71
#31 0x000056339ef6302e in main (argc=7, argv=0x7ffcc53269b8) at ./syncd/main.cpp:9

This is due to a live thread was not terminated gracefully.

what this PR does

shutdown the vpp thread gracefully.

Signed-off-by: Yue Gao <[email protected]>

mssonicbld · 2025-12-05T18:19:52Z

/azp run

azure-pipelines · 2025-12-05T18:20:03Z

Azure Pipelines successfully started running 1 pipeline(s).

kkosciusz

LGTM!

yue-fred-gao · 2025-12-08T20:23:44Z

/azpw run

mssonicbld · 2025-12-08T20:23:47Z

/AzurePipelines run

azure-pipelines · 2025-12-08T20:23:58Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2025-12-09T02:08:10Z

/azp run

azure-pipelines · 2025-12-09T02:08:20Z

Azure Pipelines successfully started running 1 pipeline(s).

yue-fred-gao · 2025-12-09T15:59:56Z

/azpw run

mssonicbld · 2025-12-09T15:59:59Z

/AzurePipelines run

azure-pipelines · 2025-12-09T16:00:22Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2025-12-10T01:17:53Z

/azp run

azure-pipelines · 2025-12-10T01:18:03Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2025-12-10T21:59:01Z

/azp run

azure-pipelines · 2025-12-10T21:59:12Z

Azure Pipelines successfully started running 1 pipeline(s).

Signed-off-by: Yue Gao <[email protected]> Signed-off-by: Vikram Menon <[email protected]>

Signed-off-by: Yue Gao <[email protected]>

* Support for sonic-mgmt ACL testing on VPP (#1599) * Enabling sonic-mgmt ACL testing for Sonic-VPP * Use retval in debugs * Graceful shutdown vpp to avoid core dump (#1714) Signed-off-by: Yue Gao <[email protected]> * vpp: support ACL attached to LAG and UDP in ACE (#1718) * Handle acl attachment through LAG update * Add default permit-all rules * Support ACL with UDP protocol * if protocol is not specified but port or port-range is create 2 rules with proto UDP and TCP. vpp requires proto to be set if port or port-range is used * realign ace stats index because each ace can map to multiple acl rules Signed-off-by: Yue Gao <[email protected]> * vpp: support binding multiple ACL tables by priority (#1732) why currently vpp doesn't support binding multiple ACL tables. Each table is appended with default permit-all rules. With multiple tables, this may cause acl matched by such rules and skip the actual rule to make in the tables after this one. what this PR does remove the default permit-all rules for each table If a table is empty, create a dummy rule that won't match any traffic because vpp doesn't allow empty table. The dummy rule matches dest-ip to 0.0.0.0/32 sort all the tables by priority in the table group. vpp doesn't support parallel matching added catch-all acl group to the end. vpp default behavior of no match is drop but sonic is accept. Fix sonic-vpp crashing due to race condition during stats pull. If the interface to get stats has been removed, stat_segment_ls_r returns null. Signed-off-by: Yue Gao <[email protected]> * changes for vpp release 202510 --------- Signed-off-by: Yue Gao <[email protected]> Co-authored-by: AkeelAli <[email protected]> Co-authored-by: Mihut Aronovici <[email protected]>

Graceful shutdown vpp to avoid core dump

7e7e415

Signed-off-by: Yue Gao <[email protected]>

kkosciusz approved these changes Dec 8, 2025

View reviewed changes

Merge branch 'master' into vpp_graceful_shut

7413fe2

Merge branch 'master' into vpp_graceful_shut

7b8d7e5

saiarcot895 approved these changes Dec 10, 2025

View reviewed changes

Merge branch 'master' into vpp_graceful_shut

1d61d64

yue-fred-gao merged commit 3ab2f07 into sonic-net:master Dec 11, 2025
16 checks passed

vikram-nexthop pushed a commit to nexthop-ai/sonic-sairedis that referenced this pull request Dec 12, 2025

Graceful shutdown vpp to avoid core dump (sonic-net#1714)

998fe41

Signed-off-by: Yue Gao <[email protected]> Signed-off-by: Vikram Menon <[email protected]>

croos12 pushed a commit to croos12/sonic-sairedis that referenced this pull request Jan 14, 2026

Graceful shutdown vpp to avoid core dump (sonic-net#1714)

d96e4d6

Signed-off-by: Yue Gao <[email protected]>

yue-fred-gao added the Request for 202511 Branch label Mar 2, 2026

yue-fred-gao added a commit to yue-fred-gao/sonic-sairedis that referenced this pull request Mar 3, 2026

Graceful shutdown vpp to avoid core dump (sonic-net#1714)

78ba5f0

Signed-off-by: Yue Gao <[email protected]>

yue-fred-gao added a commit that referenced this pull request Mar 3, 2026

Graceful shutdown vpp to avoid core dump (#1714)

c410453

Signed-off-by: Yue Gao <[email protected]>

Conversation

yue-fred-gao commented Dec 5, 2025 • edited by saiarcot895 Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

why

what this PR does

Uh oh!

mssonicbld commented Dec 5, 2025

Uh oh!

azure-pipelines bot commented Dec 5, 2025

Uh oh!

kkosciusz left a comment

Choose a reason for hiding this comment

Uh oh!

yue-fred-gao commented Dec 8, 2025

Uh oh!

mssonicbld commented Dec 8, 2025

Uh oh!

azure-pipelines bot commented Dec 8, 2025

Uh oh!

mssonicbld commented Dec 9, 2025

Uh oh!

azure-pipelines bot commented Dec 9, 2025

Uh oh!

yue-fred-gao commented Dec 9, 2025

Uh oh!

mssonicbld commented Dec 9, 2025

Uh oh!

azure-pipelines bot commented Dec 9, 2025

Uh oh!

mssonicbld commented Dec 10, 2025

Uh oh!

azure-pipelines bot commented Dec 10, 2025

Uh oh!

mssonicbld commented Dec 10, 2025

Uh oh!

azure-pipelines bot commented Dec 10, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

yue-fred-gao commented Dec 5, 2025 •

edited by saiarcot895

Loading