Leakage of context errors can tear down a service #77
Description
This is an unexpected behaviour which we discovered:
- Service is doing an action with a timeout
- That timeout happens and a function returns
context.DeadlineExceeded - That got bubbled up to
Serve()'s return value - The supervisor then treated that
context.DeadlineExceededas a service quit
I think the reasoning behind this is to handle graceful shutdown when a service stops because of a supervisor request; but the rest of the time it's surprising behaviour and a bit of a footgun.
I wonder if Suture should only treat those errors as service terminations if the service's context is canceled?
For our part we're probably going to work around this with some wrapper logic along the lines of
func (s *service) Serve(ctx context.Context) error {
err := s.Service.Serve(ctx)
if ctxErr := ctx.Err(); ctxErr != nil {
return context.Cause(ctx)
}
if errors.Is(err, context.Canceled) || errors.Is(err, context.DeadlineExceeded) {
// Mask context errors which didn't come from supervisor shutdown
return fmt.Errorf("service failed with context error: %v", err)
}
return err
}