Skip to content

acquire lock before accessing shared redis context#489

Merged
kcudnik merged 12 commits intosonic-net:masterfrom
anilkpandey:master
Jul 24, 2019
Merged

acquire lock before accessing shared redis context#489
kcudnik merged 12 commits intosonic-net:masterfrom
anilkpandey:master

Conversation

@anilkpandey
Copy link
Contributor

Multiple threads in syncd access ASIC_DB through the same redis context. g_mutex is used to synchronize the access as hiredis is not thread safe.
Function try_translate_vid_to_rid(), which accesses ASIC_DB was being called unprotected. This caused one of the threads to stuck on redisBufferRead().

(gdb) bt
#0 0x00007f15301cd22d in read () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007f1537ebf6a2 in read (__nbytes=16384, __buf=0x7f152d6b0910,
__fd=) at /usr/include/x86_64-linux-gnu/bits/unistd.h:44
#2 redisBufferRead (c=0x55c450ffdde0) at hiredis.c:802
#3 0x00007f1537ebf9a0 in redisGetReply (c=0x55c450ffdde0,
reply=reply@entry=0x7f152d6b4a00) at hiredis.c:888
#4 0x00007f1537c612d0 in swss::RedisReply::RedisReply (this=0x7f152d6b4a00,
db=0x55c451001e80, command=...) at redisreply.cpp:33
#5 0x00007f1537c614c2 in swss::RedisReply::RedisReply (this=0x7f152d6b4a00,
db=, command=..., expectedType=3) at redisreply.cpp:45
#6 0x00007f1537c7efba in swss::RedisClient::hset (this=0x55c450ffd510,
key="ASIC_STATE:SAI_OBJECT_TYPE_FDB_ENTRY:{"bvid":"oid:0x26000000000e17","mac":"3C:2C:99:8B:35:BC","switch_id":"oid:0x21", '0' <repeats 12 times>, ""}", field="SAI_FDB_ENTRY_ATTR_BRIDGE_PORT_ID",
value="oid:0x3a00000000104d") at redisclient.cpp:56

…d access from different threads

syncd main thread was doing g_redisClient->hget without acquiring g_mutex, thereby causing either syncd main thread or notification thread to block indefinitely on redisBufferRead().
@prsunny prsunny requested a review from kcudnik July 18, 2019 23:22
Copy link
Collaborator

@kcudnik kcudnik left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good catch, this is due to move processFlexCounterEvent to be executed to separate theread
which is result of making workaround for postpone counters subscribe which should be synchronous on first place

@lguohan
Copy link
Contributor

lguohan commented Jul 19, 2019

@anilkpandey, can you resolve the conflict?

@anilkpandey
Copy link
Contributor Author

@anilkpandey, can you resolve the conflict?

I am not able to, may be because I don't have write access.

@prsunny
Copy link
Contributor

prsunny commented Jul 19, 2019

You can merge/rebase from master and push to this PR. You don't have to use the resolve button above.

@prsunny
Copy link
Contributor

prsunny commented Jul 20, 2019

Looks like the editor added a lot of tab spaces. Kindly revert the space changes and recommit.

@anilkpandey
Copy link
Contributor Author

Looks like the editor added a lot of tab spaces. Kindly revert the space changes and recommit.

Done

@kcudnik kcudnik merged commit 3a8fc6f into sonic-net:master Jul 24, 2019
pettershao-ragilenetworks pushed a commit to pettershao-ragilenetworks/sonic-sairedis that referenced this pull request Nov 18, 2022
* fixed issue of permenent block on redisBufferRead() due to unprotected access from different threads

syncd main thread was doing g_redisClient->hget without acquiring g_mutex, thereby causing either syncd main thread or notification thread to block indefinitely on redisBufferRead().

* Delete .DS_Store

.

* resolved merge conflict

* Revert "resolved merge conflict"

This reverts commit f009491.

* Revert "Merge remote-tracking branch 'upstream/master'"

This reverts commit 550aca9, reversing
changes made to 676c0b5.

* Revert "Revert "Merge remote-tracking branch 'upstream/master'""

This reverts commit 03fc996.

* Revert "Revert "resolved merge conflict""

This reverts commit e840a6b.

* Update syncd.cpp

* Update syncd.cpp

* Update syncd.cpp

* Update syncd.cpp
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants