HandlerGraph.RegisterMessageType race condition

[migajek]

Under heavy load of concurrent calls, we observed some of the Requests (InvokeAsync) time out, even though the Reply is properly sent from the request handler.
For each instance started, if any of the types failed initially, they'd fail for the lifetime of the instance.

Long story short - HandlerGraph.RegisterMessageType is not thread safe.
When multiple message types are being invoked at the same time (for the first time), the registrations override each other, because of how the immutable collection is used.

Also, the issue is not self-healing since _replyTypes and _messageTypesLock go out of sync, but only _replyTypes is checked.

var graph = host.Services.GetRequiredService<HandlerGraph>();
var runtime = host.Services.GetRequiredService<IWolverineRuntime>();
Parallel.ForEach(typesToRegister, t => runtime.RegisterMessageType(t));

var missingTypes = typesToRegister.Select(t => t.ToMessageTypeName())
    .Where(t => graph.TryFindMessageType(t, out var _) is false).ToArray();

Please note - I lack ImTools experience, so I'm not sure ImTools + lock is still better choice over regular ConcurrentDictionary

[jeremydmiller]

Closed by #1668