Skip to content

Conversation

@twmb
Copy link
Owner

@twmb twmb commented Feb 2, 2026

The batchPromises ring buffer in the producer could grow unboundedly.
When records fail faster than finishPromises() can process them (e.g.,
spin-loop producing failing records), the ring would grow without bound,
potentially causing OOM within the client.

This commit rewrites the ring as a single dynamically-sized circular
buffer with optional max length blocking:

  • Buffer starts at capacity 8, grows by doubling when full
  • Shrinks back to capacity 8 when at 4 or fewer elements
  • Adds initMaxLen() to block pushes when at capacity
  • Initializes batchPromises with max(maxBufferedRecords, 8192) limit

The blocking pushes back-pressure to the application rather than OOMing
within the client.

Closes #1194

The batchPromises ring buffer in the producer could grow unboundedly.
When records fail faster than finishPromises() can process them (e.g.,
spin-loop producing failing records), the ring would grow without bound,
potentially causing OOM within the client.

This commit rewrites the ring as a single dynamically-sized circular
buffer with optional max length blocking:
- Buffer starts at capacity 8, grows by doubling when full
- Shrinks back to capacity 8 when at 4 or fewer elements
- Adds initMaxLen() to block pushes when at capacity
- Initializes batchPromises with max(maxBufferedRecords, 8192) limit

The blocking pushes back-pressure to the application rather than OOMing
within the client.

Closes #1194

Co-Authored-By: Claude Opus 4.5 <[email protected]>
@twmb twmb changed the title 1194 kgo: add backpressure to batchPromises ring buffer Feb 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add an internal limit to the batch-finishing queue so it does not grow without bound

1 participant