Skip to content

Cross-pools communication#3

Merged
umputun merged 2 commits intomasterfrom
sned
Feb 17, 2025
Merged

Cross-pools communication#3
umputun merged 2 commits intomasterfrom
sned

Conversation

@umputun
Copy link
Member

@umputun umputun commented Feb 17, 2025

This PR addresses the issue of multiple pools communicating directly with each other. It was possible to communicate safely via Collector, but this is a cumbersome way and not really what the collector is for (i.e., to collect the final result in a thread-safe way).

A much simpler flow would be if one pool worker could send data to another pool directly (see examples/direct_chain). However, to make this possible, two changes are required:

  1. Add the ability to submit data from multiple goroutines to a pool. The current Submit is not safe for such usage as it is optimized for speed, and in a typical use case, a single producer does the submissions. To address this limitation, the new Send method was added, which is essentially the same as Submit wrapped with a mutex.
  2. Add a pool-level completion callback. This is needed because we have to call the pool's Close to indicate "all data sent in." We know this on the producer side, but with multiple pools, the only way to do it efficiently is to call Close of the next pool from its completion function.

Additionally, a bunch of tests and a new example for pool-to-pool chain flow have been added.

@umputun umputun merged commit 11b9181 into master Feb 17, 2025
2 checks passed
@umputun umputun deleted the sned branch February 17, 2025 20:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant