Fix engine crash on module client blocking during keyspace events #1819

yairgott · 2025-03-05T02:27:52Z

This change enhances user experience and consistency by allowing a module to block a client on keyspace event notifications. Consistency is improved by allowing that reads after writes on the same connection yield expected results. For example, in ValkeySearch, mutations processed earlier on the same connection will be available for search.

The implementation extends VM_BlockClient to support blocking clients on keyspace event notifications. Internal clients, LUA clients, clients issueing multi exec and those with the deny_blocking flag set are not blocked. Once blocked, a client’s reply is withheld until it is explicitly unblocked.

codecov · 2025-03-05T07:23:43Z

Codecov Report

Attention: Patch coverage is 81.64557% with 29 lines in your changes missing coverage. Please review.

Project coverage is 71.03%. Comparing base (3f6581b) to head (95b4ca8).
Report is 59 commits behind head on unstable.

Files with missing lines	Patch %	Lines
src/module.c	10.00%	27 Missing ⚠️
src/networking.c	95.00%	2 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##           unstable    #1819      +/-   ##
============================================
+ Coverage     70.99%   71.03%   +0.04%     
============================================
  Files           123      123              
  Lines         65651    66033     +382     
============================================
+ Hits          46609    46909     +300     
- Misses        19042    19124      +82

Files with missing lines	Coverage Δ
src/adlist.c	`77.50% <ø> (+0.57%)`	⬆️
src/bitops.c	`94.07% <100.00%> (+0.01%)`	⬆️
src/expire.c	`96.59% <100.00%> (ø)`
src/module.h	`0.00% <ø> (ø)`
src/notify.c	`97.22% <100.00%> (+0.20%)`	⬆️
src/server.c	`87.58% <100.00%> (+0.01%)`	⬆️
src/server.h	`100.00% <ø> (ø)`
src/t_hash.c	`96.19% <100.00%> (-0.04%)`	⬇️
src/t_list.c	`92.88% <100.00%> (+0.32%)`	⬆️
src/t_set.c	`97.50% <100.00%> (-0.35%)`	⬇️
... and 5 more

... and 28 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

tests/modules/block_keyspace_notification.c

tests/unit/moduleapi/block_keyspace_notification.tcl

src/t_zset.c

src/module.c

src/networking.c

tests/unit/moduleapi/block_keyspace_notification.tcl

src/db.c

src/module.c

tests/unit/moduleapi/block_keyspace_notification.tcl

…lkey-io#1819) This change enhances user experience and consistency by allowing a module to block a client on keyspace event notifications. Consistency is improved by allowing that reads after writes on the same connection yield expected results. For example, in ValkeySearch, mutations processed earlier on the same connection will be available for search. The implementation extends `VM_BlockClient` to support blocking clients on keyspace event notifications. Internal clients, LUA clients, clients issueing multi exec and those with the `deny_blocking` flag set are not blocked. Once blocked, a client’s reply is withheld until it is explicitly unblocked. --------- Signed-off-by: yairgott <[email protected]> Signed-off-by: Jacob Murphy <[email protected]>

…lkey-io#1819) This change enhances user experience and consistency by allowing a module to block a client on keyspace event notifications. Consistency is improved by allowing that reads after writes on the same connection yield expected results. For example, in ValkeySearch, mutations processed earlier on the same connection will be available for search. The implementation extends `VM_BlockClient` to support blocking clients on keyspace event notifications. Internal clients, LUA clients, clients issueing multi exec and those with the `deny_blocking` flag set are not blocked. Once blocked, a client’s reply is withheld until it is explicitly unblocked. --------- Signed-off-by: yairgott <[email protected]> Signed-off-by: Nitai Caro <[email protected]>

…lkey-io#1819) This change enhances user experience and consistency by allowing a module to block a client on keyspace event notifications. Consistency is improved by allowing that reads after writes on the same connection yield expected results. For example, in ValkeySearch, mutations processed earlier on the same connection will be available for search. The implementation extends `VM_BlockClient` to support blocking clients on keyspace event notifications. Internal clients, LUA clients, clients issueing multi exec and those with the `deny_blocking` flag set are not blocked. Once blocked, a client’s reply is withheld until it is explicitly unblocked. --------- Signed-off-by: yairgott <[email protected]>

yairgott · 2025-04-22T20:00:38Z

This test failed in Daily in the reply-schemas-validator job:

https://github.com/valkey-io/valkey/actions/runs/14554113682/job/40828749380#step:7:627

https://github.com/valkey-io/valkey/actions/runs/14543642514/job/40805916104#step:7:626

It's an assert logreqres.c:263 'ret' is not true and the stack trace is a module unblocking a client.

It happens in the test case "Blocking keyspace notification one".

Could be. I suspect that the following should be moved to line 501. I'll give it a try.

yairgott · 2025-04-22T20:56:10Z

This test failed in Daily in the reply-schemas-validator job:

https://github.com/valkey-io/valkey/actions/runs/14554113682/job/40828749380#step:7:627

https://github.com/valkey-io/valkey/actions/runs/14543642514/job/40805916104#step:7:626

It's an assert logreqres.c:263 'ret' is not true and the stack trace is a module unblocking a client.
It happens in the test case "Blocking keyspace notification one".

Could be. I suspect that the following should be moved to line 501. I'll give it a try.

Yeah, it fixed it. See #1989

See #1819 (comment) Verified the fix by: 1. Build: `make SERVER_CFLAGS='-Werror -DLOG_REQ_RES' -j10` 2. Running module api tests: `CFLAGS='-Werror' ./runtest-moduleapi --log-req-res --no-latency --dont-clean --force-resp3 --dont-pre-clean --verbose --dump-logs` --------- Signed-off-by: yairgott <[email protected]>

…lkey-io#1819) This change enhances user experience and consistency by allowing a module to block a client on keyspace event notifications. Consistency is improved by allowing that reads after writes on the same connection yield expected results. For example, in ValkeySearch, mutations processed earlier on the same connection will be available for search. The implementation extends `VM_BlockClient` to support blocking clients on keyspace event notifications. Internal clients, LUA clients, clients issueing multi exec and those with the `deny_blocking` flag set are not blocked. Once blocked, a client’s reply is withheld until it is explicitly unblocked. --------- Signed-off-by: yairgott <[email protected]> Signed-off-by: Jacob Murphy <[email protected]>

See valkey-io#1819 (comment) Verified the fix by: 1. Build: `make SERVER_CFLAGS='-Werror -DLOG_REQ_RES' -j10` 2. Running module api tests: `CFLAGS='-Werror' ./runtest-moduleapi --log-req-res --no-latency --dont-clean --force-resp3 --dont-pre-clean --verbose --dump-logs` --------- Signed-off-by: yairgott <[email protected]> Signed-off-by: Jacob Murphy <[email protected]>

) This change enhances user experience and consistency by allowing a module to block a client on keyspace event notifications. Consistency is improved by allowing that reads after writes on the same connection yield expected results. For example, in ValkeySearch, mutations processed earlier on the same connection will be available for search. The implementation extends `VM_BlockClient` to support blocking clients on keyspace event notifications. Internal clients, LUA clients, clients issueing multi exec and those with the `deny_blocking` flag set are not blocked. Once blocked, a client’s reply is withheld until it is explicitly unblocked. --------- Signed-off-by: yairgott <[email protected]> Signed-off-by: Jacob Murphy <[email protected]>

See #1819 (comment) Verified the fix by: 1. Build: `make SERVER_CFLAGS='-Werror -DLOG_REQ_RES' -j10` 2. Running module api tests: `CFLAGS='-Werror' ./runtest-moduleapi --log-req-res --no-latency --dont-clean --force-resp3 --dont-pre-clean --verbose --dump-logs` --------- Signed-off-by: yairgott <[email protected]> Signed-off-by: Jacob Murphy <[email protected]>

…lkey-io#1819) This change enhances user experience and consistency by allowing a module to block a client on keyspace event notifications. Consistency is improved by allowing that reads after writes on the same connection yield expected results. For example, in ValkeySearch, mutations processed earlier on the same connection will be available for search. The implementation extends `VM_BlockClient` to support blocking clients on keyspace event notifications. Internal clients, LUA clients, clients issueing multi exec and those with the `deny_blocking` flag set are not blocked. Once blocked, a client’s reply is withheld until it is explicitly unblocked. --------- Signed-off-by: yairgott <[email protected]> Signed-off-by: hwware <[email protected]>

See valkey-io#1819 (comment) Verified the fix by: 1. Build: `make SERVER_CFLAGS='-Werror -DLOG_REQ_RES' -j10` 2. Running module api tests: `CFLAGS='-Werror' ./runtest-moduleapi --log-req-res --no-latency --dont-clean --force-resp3 --dont-pre-clean --verbose --dump-logs` --------- Signed-off-by: yairgott <[email protected]> Signed-off-by: hwware <[email protected]>

See valkey-io#1819 (comment) Verified the fix by: 1. Build: `make SERVER_CFLAGS='-Werror -DLOG_REQ_RES' -j10` 2. Running module api tests: `CFLAGS='-Werror' ./runtest-moduleapi --log-req-res --no-latency --dont-clean --force-resp3 --dont-pre-clean --verbose --dump-logs` --------- Signed-off-by: yairgott <[email protected]>

src/module.c

This flag.deny_blocking check causes some code to becomde dead code. Some of the code below checks islua or ismulti, but they are marked with flag.deny_blocking and will return early. In addition, cleanup some documents, some of them are inaccurate, and restore the code of blocking_auth_cb in tests/modules/auth.c, this reply should be returned from core. The reply used to from the core and was changed to from the module in valkey-io#1819, and now it is from the core again. Cleanup some dead code around valkey-io#1819. Signed-off-by: Binbin <[email protected]>

When we add f8a5a4f, a module auth can add reply in its callback, or not add reply and just return an error robj in its callback, and we have to figure a way to determine whether it has added the reply. Before we are using `clientHasPendingReplies` and it had a issue, the auth might be used inside a transaction or pipeline which already has some pending replies in the client reply list. It causes us to not add the error reply to auth. After valkey-io#1819, now we have a buffered_reply client flag, it indicates the reply for the current command was buffered, either in client::reply or in client::buf. We can use this flag to check. Fixes valkey-io#2106. Signed-off-by: Binbin <[email protected]>

When we add f8a5a4f, a module auth can add reply in its callback, or not add reply and just return an error robj in its callback, and we have to figure a way to determine whether it has added the reply. Before we are using `clientHasPendingReplies` and it had a issue, the auth might be used inside a transaction or pipeline which already has some pending replies in the client reply list. It causes us to not add the error reply to auth. After #1819, now we have a buffered_reply client flag, it indicates the reply for the current command was buffered, either in client::reply or in client::buf. We can use this flag to check. Fixes #2106. --------- Signed-off-by: Binbin <[email protected]>

…y-io#2287) When we add f8a5a4f, a module auth can add reply in its callback, or not add reply and just return an error robj in its callback, and we have to figure a way to determine whether it has added the reply. Before we are using `clientHasPendingReplies` and it had a issue, the auth might be used inside a transaction or pipeline which already has some pending replies in the client reply list. It causes us to not add the error reply to auth. After valkey-io#1819, now we have a buffered_reply client flag, it indicates the reply for the current command was buffered, either in client::reply or in client::buf. We can use this flag to check. Fixes valkey-io#2106. --------- Signed-off-by: Binbin <[email protected]>

…#2215) This flag.deny_blocking check causes some code to becomde dead code. Some of the code below checks islua or ismulti, but they are marked with flag.deny_blocking and will return early. In addition, cleanup some documents, some of them are inaccurate, and restore the code of blocking_auth_cb in tests/modules/auth.c, this reply should be returned from core. The reply used to from the core and was changed to from the module in #1819, and now it is from the core again. Cleanup some dead code around #1819. Signed-off-by: Binbin <[email protected]>

…valkey-io#2215) This flag.deny_blocking check causes some code to becomde dead code. Some of the code below checks islua or ismulti, but they are marked with flag.deny_blocking and will return early. In addition, cleanup some documents, some of them are inaccurate, and restore the code of blocking_auth_cb in tests/modules/auth.c, this reply should be returned from core. The reply used to from the core and was changed to from the module in valkey-io#1819, and now it is from the core again. Cleanup some dead code around valkey-io#1819. Signed-off-by: Binbin <[email protected]>

…#2215) This flag.deny_blocking check causes some code to becomde dead code. Some of the code below checks islua or ismulti, but they are marked with flag.deny_blocking and will return early. In addition, cleanup some documents, some of them are inaccurate, and restore the code of blocking_auth_cb in tests/modules/auth.c, this reply should be returned from core. The reply used to from the core and was changed to from the module in #1819, and now it is from the core again. Cleanup some dead code around #1819. Signed-off-by: Binbin <[email protected]>

…valkey-io#2215) This flag.deny_blocking check causes some code to becomde dead code. Some of the code below checks islua or ismulti, but they are marked with flag.deny_blocking and will return early. In addition, cleanup some documents, some of them are inaccurate, and restore the code of blocking_auth_cb in tests/modules/auth.c, this reply should be returned from core. The reply used to from the core and was changed to from the module in valkey-io#1819, and now it is from the core again. Cleanup some dead code around valkey-io#1819. Signed-off-by: Binbin <[email protected]> Signed-off-by: Harkrishn Patro <[email protected]>

yairgott force-pushed the client_block branch from 150faea to 79f435d Compare March 5, 2025 02:28

yairgott changed the title ~~Enable Blocking a Client on Keyspace Event Notification~~ Support Blocking a Client on Keyspace Event Notification Mar 5, 2025

yairgott force-pushed the client_block branch 4 times, most recently from a064f39 to 74cb5e4 Compare March 5, 2025 07:08

PingXie requested a review from enjoy-binbin March 6, 2025 00:44

enjoy-binbin reviewed Mar 6, 2025

View reviewed changes

tests/modules/block_keyspace_notification.c Show resolved Hide resolved

enjoy-binbin reviewed Mar 6, 2025

View reviewed changes

tests/unit/moduleapi/block_keyspace_notification.tcl Outdated Show resolved Hide resolved

enjoy-binbin reviewed Mar 6, 2025

View reviewed changes

src/t_zset.c Outdated Show resolved Hide resolved

yairgott force-pushed the client_block branch from 74cb5e4 to 08faaef Compare March 6, 2025 04:21

madolson reviewed Mar 6, 2025

View reviewed changes

src/module.c Outdated Show resolved Hide resolved

src/networking.c Outdated Show resolved Hide resolved

tests/unit/moduleapi/block_keyspace_notification.tcl Show resolved Hide resolved

QuChen88 reviewed Mar 6, 2025

View reviewed changes

src/db.c Outdated Show resolved Hide resolved

QuChen88 reviewed Mar 6, 2025

View reviewed changes

src/module.c Outdated Show resolved Hide resolved

QuChen88 reviewed Mar 6, 2025

View reviewed changes

src/module.c Outdated Show resolved Hide resolved

QuChen88 reviewed Mar 6, 2025

View reviewed changes

src/module.c Show resolved Hide resolved

yairgott force-pushed the client_block branch 3 times, most recently from 2428ae0 to 7c9cff2 Compare March 6, 2025 23:06

madolson reviewed Mar 7, 2025

View reviewed changes

tests/unit/moduleapi/block_keyspace_notification.tcl Outdated Show resolved Hide resolved

yairgott force-pushed the client_block branch 9 times, most recently from dd60c6a to 0445d46 Compare March 7, 2025 22:00

zuiderkwast moved this from To be backported to 8.1.1 in Valkey 8.1 Apr 22, 2025

yairgott deleted the client_block branch April 22, 2025 20:00

yairgott mentioned this pull request Apr 22, 2025

blocking client followup fix related to log_req_res #1989

Merged

enjoy-binbin reviewed Jun 15, 2025

View reviewed changes

src/module.c Show resolved Hide resolved

enjoy-binbin reviewed Jun 15, 2025

View reviewed changes

src/module.c Show resolved Hide resolved

enjoy-binbin mentioned this pull request Jun 15, 2025

Remove deny_blocking check in moduleBlockClient, cleanup code and doc #2215

Merged

enjoy-binbin mentioned this pull request Jun 30, 2025

Fix missing response when AUTH is errored inside a transaction #2287

Merged

enjoy-binbin mentioned this pull request Nov 5, 2025

Fix memory leak in deferred reply buffer #2615

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Fix engine crash on module client blocking during keyspace events #1819

Fix engine crash on module client blocking during keyspace events #1819

Uh oh!

yairgott commented Mar 5, 2025 •

edited

Loading

Uh oh!

codecov bot commented Mar 5, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

yairgott commented Apr 22, 2025

Uh oh!

yairgott commented Apr 22, 2025

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Fix engine crash on module client blocking during keyspace events #1819

Fix engine crash on module client blocking during keyspace events #1819

Uh oh!

Conversation

yairgott commented Mar 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Mar 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

yairgott commented Apr 22, 2025

Uh oh!

yairgott commented Apr 22, 2025

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

yairgott commented Mar 5, 2025 •

edited

Loading

codecov bot commented Mar 5, 2025 •

edited

Loading