Skip to content

Improve RabbitMQ connection retry code #4509

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 4 commits into from
Jan 22, 2019
Merged

Improve RabbitMQ connection retry code #4509

merged 4 commits into from
Jan 22, 2019

Conversation

@Kami
Copy link
Member

@Kami Kami commented Jan 22, 2019

This pull request includes two improvements to our RabbitMQ connection retry code:

  1. Fix the code so we don't try to re-establish a connection for indefinitely in a place where we are calling connection.ensure_connection. Previously we didn't pass max_retries argument to that method so the connection was tried to be re-established for indefinitely and we didn't log any error.
  2. Log an error which is received when failing to re-establish a connection on each retry attempt. This is intentionally logged as DEBUG because it's only fatal if we exhaust all the retry attempts and still fail to re-establish a connection (in that case we re-throw the exception and issue is considered as fatal).

This should help and give more insight in #4508 (it's likely that RabbitMQ server is not running there or similar).

@Kami
Fix RabbitMQ connection retry wrapper so it doesn't block for
1d11897 
indefinetly if connection to RabbitMQ cannot be established.

Also add errback which logs an actual error when trying to re-establish
a connection.
@Kami Kami added the bug label Jan 22, 2019
@Kami Kami added this to the 3.0.0 milestone Jan 22, 2019
@Kami Kami mentioned this pull request Jan 22, 2019
Closed
@Kami Kami requested a review from bigmstone January 22, 2019 12:49
bigmstone
bigmstone approved these changes Jan 22, 2019
View reviewed changes
Copy link
Contributor

@bigmstone bigmstone left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@Kami Kami merged commit a588698 into master Jan 22, 2019
@Kami Kami deleted the update_rabbitmq_connection_retry_code branch January 22, 2019 17:43
@Kami Kami modified the milestones: 3.0.0, 2.10.2 Jan 24, 2019
@Kami Kami mentioned this pull request Feb 18, 2019
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer

@bigmstone bigmstone bigmstone approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

bug rabbitmq

Projects

None yet

Milestone

2.10.2

Development

Successfully merging this pull request may close these issues.

3 participants

@Kami @bigmstone