Add possibility to compile applications with address sanitizer. #2123
Closed
oleksandrivantsiv wants to merge 4 commits intosonic-net:masterfrom
Closed
Add possibility to compile applications with address sanitizer. #2123oleksandrivantsiv wants to merge 4 commits intosonic-net:masterfrom
oleksandrivantsiv wants to merge 4 commits intosonic-net:masterfrom
Conversation
Add "--enable-asan" configure option to enable address sanitizer for all SWSS applications. Add possibility to trigger applications to exit by handling SIGTERM signal. Initialize ASAN default options for all applications. Signed-off-by: Oleksandr Ivantsiv <[email protected]>
Signed-off-by: Oleksandr Ivantsiv <[email protected]>
Signed-off-by: Oleksandr Ivantsiv <[email protected]>
oleksandrivantsiv
requested review from
dgsudharsan,
kcudnik,
lguohan and
qiluo-msft
| VTEPTable vtep_table_; | ||
| Table m_stateVxlanTable; | ||
| std::map<sai_object_id_t, std::string> m_pendingAddToFlexCntr; | ||
| FlexCounterManager vxlan_tunnel_stat_manager; |
Collaborator
Author
There was a problem hiding this comment.
ASAN report for the issue:
=================================================================
==45==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000000 (pc 0x557cc1c61f04 bp 0x6030000c5dd0 sp 0x7fffc0c41f40 T0)
==45==The signal is caused by a READ memory access.
==45==Hint: address points to the zero page.
#0 0x557cc1c61f03 in FlexCounterManager::~FlexCounterManager() flex_counter/flex_counter_manager.cpp:116
#1 0x557cc19e3326 in VxlanTunnelOrch::~VxlanTunnelOrch() /sonic/src/sonic-swss/orchagent/vxlanorch.h:259
#2 0x557cc19e3326 in VxlanTunnelOrch::~VxlanTunnelOrch() /sonic/src/sonic-swss/orchagent/vxlanorch.h:259
#3 0x557cc137d400 in OrchDaemon::~OrchDaemon() /sonic/src/sonic-swss/orchagent/orchdaemon.cpp:86
#4 0x557cc1330ac6 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() /usr/include/c++/8/bits/shared_ptr_base.h:155
#5 0x557cc1330ac6 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() /usr/include/c++/8/bits/shared_ptr_base.h:148
#6 0x557cc12323fb in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count() /usr/include/c++/8/bits/shared_ptr_base.h:728
#7 0x557cc12323fb in std::__shared_ptr<OrchDaemon, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr() /usr/include/c++/8/bits/shared_ptr_base.h:1167
#8 0x557cc12323fb in std::shared_ptr<OrchDaemon>::~shared_ptr() /usr/include/c++/8/bits/shared_ptr.h:103
#9 0x557cc12323fb in main /sonic/src/sonic-swss/orchagent/main.cpp:675
#10 0x7f166fcdf09a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2409a)
#11 0x557cc1328819 in _start (/usr/bin/orchagent+0x310819)
AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV flex_counter/flex_counter_manager.cpp:116 in FlexCounterManager::~FlexCounterManager()
==45==ABORTING
| * when iterating ConsumerMap. This is ensured implicitly by the order of keys in ordered map. | ||
| * For cases when Orch has to process tables in specific order, like PortsOrch during warm start, it has to override Orch::doTask() | ||
| */ | ||
| m_orchList = { gSwitchOrch, gCrmOrch, gPortsOrch, gBufferOrch, mux_orch, mux_cb_orch, gIntfsOrch, gNeighOrch, gNhgMapOrch, gNhgOrch, gCbfNhgOrch, gRouteOrch, gCoppOrch, qos_orch, wm_orch, policer_orch, tunnel_decap_orch, sflow_orch, debug_counter_orch, gMacsecOrch, gBfdOrch, gSrv6Orch}; |
Collaborator
Author
There was a problem hiding this comment.
ASAN report for the issue:
=================================================================
==45==ERROR: AddressSanitizer: heap-use-after-free on address 0x617000004b20 at pc 0x5566372c5a51 bp 0x7fff744ff6c0 sp 0x7fff744ff6b8
READ of size 8 at 0x617000004b20 thread T0
#0 0x5566372c5a50 in NeighOrch::~NeighOrch() /sonic/src/sonic-swss/orchagent/neighorch.cpp:56
#1 0x5566372c5a88 in NeighOrch::~NeighOrch() /sonic/src/sonic-swss/orchagent/neighorch.cpp:58
#2 0x556637169400 in OrchDaemon::~OrchDaemon() /sonic/src/sonic-swss/orchagent/orchdaemon.cpp:86
#3 0x55663711cac6 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() /usr/include/c++/8/bits/shared_ptr_base.h:155
#4 0x55663711cac6 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() /usr/include/c++/8/bits/shared_ptr_base.h:148
#5 0x55663701e3fb in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count() /usr/include/c++/8/bits/shared_ptr_base.h:728
#6 0x55663701e3fb in std::__shared_ptr<OrchDaemon, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr() /usr/include/c++/8/bits/shared_ptr_base.h:1167
#7 0x55663701e3fb in std::shared_ptr<OrchDaemon>::~shared_ptr() /usr/include/c++/8/bits/shared_ptr.h:103
#8 0x55663701e3fb in main /sonic/src/sonic-swss/orchagent/main.cpp:675
#9 0x7f2e0320b09a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2409a)
#10 0x556637114819 in _start (/usr/bin/orchagent+0x310819)
0x617000004b20 is located 288 bytes inside of 720-byte region [0x617000004a00,0x617000004cd0)
freed by thread T0 here:
#0 0x7f2e03d9b128 in operator delete(void*, unsigned long) (/usr/lib/x86_64-linux-gnu/libasan.so.5+0xec128)
#1 0x556637574384 in FdbOrch::~FdbOrch() /sonic/src/sonic-swss/orchagent/fdborch.h:90
#2 0x556637c5fe39 (/usr/bin/orchagent+0xe5be39)
previously allocated by thread T0 here:
#0 0x7f2e03d99d30 in operator new(unsigned long) (/usr/lib/x86_64-linux-gnu/libasan.so.5+0xead30)
#1 0x5566371735c8 in OrchDaemon::init() /sonic/src/sonic-swss/orchagent/orchdaemon.cpp:129
#2 0x55663701e39c in main /sonic/src/sonic-swss/orchagent/main.cpp:689
SUMMARY: AddressSanitizer: heap-use-after-free /sonic/src/sonic-swss/orchagent/neighorch.cpp:56 in NeighOrch::~NeighOrch()
Shadow bytes around the buggy address:
0x0c2e7fff8910: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c2e7fff8920: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c2e7fff8930: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c2e7fff8940: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c2e7fff8950: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
=>0x0c2e7fff8960: fd fd fd fd[fd]fd fd fd fd fd fd fd fd fd fd fd
0x0c2e7fff8970: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c2e7fff8980: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c2e7fff8990: fd fd fd fd fd fd fd fd fd fd fa fa fa fa fa fa
0x0c2e7fff89a0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c2e7fff89b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Shadow byte legend (one shadow byte represents 8 application bytes):
Addressable: 00
Partially addressable: 01 02 03 04 05 06 07
Heap left redzone: fa
Freed heap region: fd
Stack left redzone: f1
Stack mid redzone: f2
Stack right redzone: f3
Stack after return: f5
Stack use after scope: f8
Global redzone: f9
Global init order: f6
Poisoned by user: f7
Container overflow: fc
Array cookie: ac
Intra object redzone: bb
ASan internal: fe
Left alloca redzone: ca
Right alloca redzone: cb
==45==ABORTING
Collaborator
Author
|
ASAN orchagent memory leaks reports: |
dgsudharsan
previously approved these changes
Jan 26, 2022
Collaborator
|
/azp run Azure.sonic-swss |
|
Azure Pipelines successfully started running 1 pipeline(s). |
Collaborator
|
/azp run Azure.sonic-swss |
|
Azure Pipelines successfully started running 1 pipeline(s). |
libasan should be the first library to link with, to allow it to inspect others. Signed-off-by: Oleksandr Ivantsiv <[email protected]>
dgsudharsan
approved these changes
Feb 9, 2022
Collaborator
|
/azp run Azure.sonic-swss |
|
Azure Pipelines successfully started running 1 pipeline(s). |
Collaborator
|
@oleksandrivantsiv can you please update your branch to the latest and check failures? |
Collaborator
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
Collaborator
|
@oleksandrivantsiv please handle conflicts and rerun checkers. |
Contributor
|
please resolve conflicts |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add SIGTERM handling as a trigger to exit to all applications.
Fix "memory usage after free" error reported by ASAN.
Fix "SEGV on unknown address" error reported by ASAN.
Fix orchagent memory leaks reported by ASAN.
What I did
To add a possibility to compile SONiC applications with address sanitizer (ASAN).
Add SIGTERM handling as a trigger to exit to all applications. The correct shutdown of applications is needed to allow ASAN to generate a report.
Why I did it
To add a possibility to detect memory errors in applications.
How I verified it
Compile SWSS with --enable-asan option. Install applications to DUT. Run tests that trigger different kinds of memory usage inside of applications. Stop applications with SIGTERM signal to trigger ASAN report generation. Reports will be placed in /var/log/asan/ directory.
Details if related
Depends on:
sonic-net/sonic-buildimage#9857
sonic-net/sonic-swss-common#579