AsyncProducer Close blocked for infinity #2121
Description
Versions
Please specify real version numbers or git SHAs, not just "Latest" since that changes fairly regularly.
| Sarama | Kafka | Go |
|---|---|---|
| 1.27.2 | 3.0.0 | 1.7.5 |
Configuration
What configuration values are you using for Sarama and Kafka?
config.Metadata.Retry.Max = 3
config.Metadata.Retry.Backoff = 250 * time.Millisecond
config.Metadata.Timeout = 1 * time.Minute
// Admin.Retry take effect on `ClusterAdmin` related operations,
// only `CreateTopic` for cdc now. Just use default values.
config.Admin.Retry.Max = 5
config.Admin.Retry.Backoff = 100 * time.Millisecond
config.Admin.Timeout = 3 * time.Second
config.Producer.Retry.Max = 3
config.Producer.Retry.Backoff = 100 * time.Millisecond
config.Producer.Partitioner = sarama.NewManualPartitioner
config.Producer.MaxMessageBytes = c.MaxMessageBytes
config.Producer.Return.Successes = true
config.Producer.Return.Errors = true
config.Producer.RequiredAcks = sarama.WaitForAllLogs
We are testing sarama in an extremely rare scenario:
- producer can send request to the 1 machine Kafka cluster, but cannot get a response
Kill -s STOPthe kafka broker process, this will make the TCP connection remain, but will not send a response.- Close the producer by call
asyncProducer.Close()
we do not have a log from sarama, but have grabbed some goroutine stack, like the following one, it looks blocked on trying to receive a response from the broker.
Our purpose is that when try to close the producer, it should not be blocked for a long time, instead of return as soon as possible.
But the reality as shown in the picture above, 33 messages failed to deliver after 38minutes, and it was after the process resume by kill -s CONT