[KIP-932] Skip offset filter for share consumers in msgset parser#5354
Merged
Pranav Rathi (pranavrth) merged 1 commit intodev_kip-932_queues-for-kafkafrom Mar 13, 2026
Conversation
|
🎉 All Contributor License Agreements have been signed. Ready to merge. |
airlock-confluentinc
bot
force-pushed
the
dev_kip-932_remove-fetch-pos-from-message-parsing
branch
from
e9cf84e to
8ef7362
Compare
…parser Share consumers may receive re-delivered records at earlier offsets after RELEASE. The offset checks in the msgset parser that skip messages where offset < fetch_pos.offset are correct for regular consumers (monotonically increasing offsets) but cause share consumers to drop re-delivered records, resulting in spurious GAP acks. 1. Add RD_KAFKA_IS_SHARE_CONSUMER guard to all four offset checks (msg_v0_1, msg_v2, msgset_reader_v2, peek_msg_version) so share consumers never skip messages based on offset position. 2. Add TODO comments for future investigation of fetch_pos.offset handling in share consumers (postproc relative offset fixup, fetch_pos update after message delivery, and next_offset outlier adjustment).
airlock-confluentinc
bot
force-pushed
the
dev_kip-932_remove-fetch-pos-from-message-parsing
branch
from
8ef7362 to
9600c85
Compare
Pratyush Ranjan (PratRanj07)
requested changes
Mar 13, 2026
| @@ -653,7 +653,8 @@ rd_kafka_msgset_reader_msg_v0_1(rd_kafka_msgset_reader_t *msetr) { | |||
| * we cant perform this offset check here | |||
| * in that case. */ | |||
Contributor
There was a problem hiding this comment.
Suggested change
| * in that case. */ | |
| * in that case. | |
| * Share consumers may receive re-delivered records at earlier offsets | |
| * after RELEASE, so skip this check for them.*/ |
Contributor
There was a problem hiding this comment.
Or maybe add a TODO to remove this handling after making sure we dont recieve Messages v0 and v1 after broker v4.0
Pratyush Ranjan (PratRanj07)
approved these changes
Mar 13, 2026
Contributor
Pratyush Ranjan (PratRanj07)
left a comment
Pratyush Ranjan (PratRanj07)
left a comment
There was a problem hiding this comment.
LGTM! Thanks Pranav Rathi (@pranavrth) We can do the comment change in another PR
Pranav Rathi (pranavrth)
merged commit a380f92
into
dev_kip-932_queues-for-kafka
3 checks passed
Pranav Rathi (pranavrth)
deleted the
dev_kip-932_remove-fetch-pos-from-message-parsing
branch
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Share consumers may receive re-delivered records at earlier offsets after RELEASE. The three offset checks in the msgset parser that skip messages where offset < fetch_pos.offset are correct for regular consumers (monotonically increasing offsets) but cause share consumers to drop re-delivered records, resulting in spurious GAP acks.
Add RD_KAFKA_IS_SHARE_CONSUMER guard to all three offset checks so share consumers never skip messages based on offset position.