Ensure message pump handles rate limiting requests

[ramonsmits]

Describe the feature.

Currently when there are rate limiting failures during processing messages are being fetched immediately for processing again and the transport is not throttling to adhere to the rate limiting request

Considerations:

1. Thottle / delay processing if the error is due to rate limiting
2. Support rate limiting so that instance are not overloading Azure Service Bus

Additional Context

No response

[SeanFeldman]

Throttling by time, by # of messages, or a custom metric that is related to the number of messages but not 1 to 1.

[ramonsmits]

but not 1 to 1.

@SeanFeldman What do you mean by this?

[SeanFeldman]

For example, Intercom API allows up to 16 operations per second. But a single message might represent one or more operations against a 3rd party. Therefore, it's not always 16 messages per second but rather less. The endpoint needs to know how many operations it has performed to slow down or stop receiving until it can resume.

Another example is CosmosDB or any other resource that has throttling in place and reports back the quota of operations used or left.

[ramonsmits]

@SeanFeldman ok, rate limiting from the handlers processing perspective that is connecting to services. This issue is about rate limiting on the broker.

[SeanFeldman]

On the broker? The broker will give what it can. This is on endpoints. Endpoints pump messages.

[ramonsmits]

The broker will rate limit / throttle the namespace when consumers fetch too many messages. I'm not sure about the reason as docs mention connection limits too but the transport should gracefully handle that and not immediately fetch/reconnect when that happens.

[SeanFeldman]

That's doesn't sound right, @ramonsmits.
How would the broker know what is "too many messages"? The broker will only throttle based on your namespace tier and MUs provisioned in Premium. Nothing else. The throughput of an endpoint is an endpoint business, not the broker. Hence this issue.

Think about it, if a namespace can send you N messages that result in O operations, where N < O, the namespace is not throttling anything. It's the 3rd part that does because it cannot accept more that Max messages, where N <= Max < O.