Distinguish between expected and unexpected failures #232
Description
Today, Menes always reports exceptions that emerge from service methods. However, it does not make it easy to distinguish between exceptions that are expected (e.g., throwing an OpenApiUnauthorizedException to produce a 401 response), and ones that we simply failed to anticipate.
These two cases require quite different responses. If an exception occurs that we were not expecting, we should investigate, because that sounds like a coding error. By contrast, an expected OpenApiUnauthorizedException looks like business as usual.
It is possible to detect these cases indirectly today in Application Insights by defining an alert that watches for 500 status codes in the HTTP response. And if you then go and look at that in Application Insights, the correlation ID should enable you to see more detail about the request, including the exception that led to the 500.
However, this is a somewhat indirect mechanism. It could be defeated if Application Insights has started to sample—we might end up knowing that a problem has occurred but be entirely unable to diagnose it. So it would be better to have a clear, unambiguous indication of this kind of problem.
This might require extensions to Corvus.Monitoring—we might want to add an abstraction analogous to Application Insights custom events.