Inbox - no sequential processing

[Dark90lab]

Describe the bug

When endpoint is configured as:

endpointConfiguration.UseDurableInbox().ListenWithStrictOrdering();

and there are some unprocessed messages in the inbox before the consumer application is launched; the application does not first process messages from the inbox and then does not allow them to be processed from the transport (Kafka). After launching, the application immediately starts processing messages that arrive from the transport, only after some set time the Timer (DurabilityAgent) is started, which results in the execution of a batch of database operations, including CheckRecoverableIncomingMessagesOperation. Next, the local message processing queue is continuously filled with messages from the transport. If the queue size allows, it is also periodically filled with messages from the inbox after the recovery operation. This causes the processing of incoming messages to be out of sequence.

To Reproduce
Steps to reproduce the behavior:

1. Insert messages in state Incoming to the inbox table
2. Push some messages on transport (kafka)
3. Log information in the handler that message was processed
4. See in logs sequence of messages processing

Expected behavior

In ListenWithStrictOrdering mode, the data source for processing should be the inbox. Currently, messages are asynchronously pushed to the inbox and immediately pushed to the local processing queue. Messages consumed by the application should be processed in the order they were retrieved from the transport.

Additional context
Problem was discovered in cooperation with @dominikjeske.
Wolverine version: 4.10.1

[jeremydmiller]

@dominikjeske and @Dark90lab By no stretch of the imagination is the behavior you're wanting here something that has ever been considered or documented. I'd treat this as a large feature request to be honest.

@dominikjeske This would be beyond the scope of the support contract we've previously discussed.