outbound: implement OutboundPolicies route request timeouts

[hawkw]

The latest proxy-api release, v0.10.0, adds fields to the
OutboundPolicies API for configuring HTTP request timeouts, based on
the proposed changes to HTTPRoute in kubernetes-sigs/gateway-api#1997.

This branch updates the proxy-api dependency to v0.10.0 and adds the new
timeout configuration fields to the proxy's internal client policy
types. In addition, this branch adds a timeout middleware to the HTTP
client policy stack, so that the timeout described by the
Rule.request_timeout field is now applied.

Implementing the RouteBackend.request_timeout field with semantics as
close as possible to those described in GEP-1742 will be somewhat more
complex, and will be added in a separate PR. The reason for this
complexity is that, as described in GEP-1742, the
timeouts.backendRequest timeout is intended to bound only the time it
takes for the backend to respond to the request, rather than bounding
both time spent in the proxy and time spent in the backend (which isthe
intention behind the timeouts.request timeout). Therefore, the timeout
for backendRequest should be started as "close" to the actual backend
client as possible, ideally in the endpoint stack after the request has
already made it through the load balancer queue. Starting this timeout
in the logical stack like the route's request_timeout does would mean
that time spent in the proxy's queues is included in this timeout, which
seems not to be the intention. However, if we want these timeouts to
start in the endpoint stack, we can't configure them using the target
type, because a backend may be referenced with different timeouts by
different HTTPRoutes, and we would prefer not to create separate client
stacks based on those timeout values. Instead, we will need the request
to carry the configured timeout value as an extension, and a new timeout
middleware that applies timeouts from a request extension. This will be
implemented in a follow-up PR.

[adleong]

Are there any proxy metrics which are incremented when a timeout occurs? Are timeouts distinguishable from application errors? Are request timeouts distinguishable from connect timeouts, detect timeouts, or failfast?

[hawkw]

@adleong

Are there any proxy metrics which are incremented when a timeout occurs?

With regards to metrics, these timeouts should behave analogously to timeouts configured for ServiceProfiles. When a request times out, we return an HTTP 504 response to the application, and increment the response_total metric with that status code as a label.

Are timeouts distinguishable from application errors?

AFAIK, we don't currently have a metric label that distinguishes HTTP 504 error codes that result from internal proxy timeouts from those that are returned by an application service. It's probably a good idea to add a label or something that make proxy timeouts distinguishable from those returned by an upstream application, but this maybe deserves some additional design work --- we could probably make it possible to distinguish between any error status synthesized by the proxy and errors with the same status returned by a remote application service.

Are request timeouts distinguishable from connect timeouts, detect timeouts, or failfast?

Protocol detection timeouts don't result in an error being returned to the application; when protocol detection times out, the proxy handles the connection as opaque. This is a different behavior than returning an error, so they are distinguishable because the proxy does something different.

Regarding failfast, the proxy will currently return an HTTP 504 for requests already dispatched to a service when it enters failfast, but returns an HTTP 503 when a new request would be dispatched to a service which is shedding load due to failfast:

linkerd2-proxy/linkerd/app/outbound/src/http/server.rs

Lines 154 to 163 in 704ef31

	// No available backend can be found for a request.
	if errors::is_caused_by::<errors::FailFastError>(&*error) {
	// XXX(ver) This should probably be SERVICE_UNAVAILABLE, because
	// this is basically no different from a LoadShedError, but that
	// would be a change in behavior.
	return Ok(errors::SyntheticHttpResponse::gateway_timeout(error));
	}
	if errors::is_caused_by::<errors::LoadShedError>(&*error) {
	return Ok(errors::SyntheticHttpResponse::unavailable(error));
	}

This means both timeouts will sometimes increment the response_total metric with status="504", so they cannot always be distinguished in metrics currently.

In all cases, we will log substantially different error messages for these timeouts, and the value of the l5d-proxy-error header will be different. So these timeouts can be distinguished that way.

[hawkw]

@adleong Update regarding my previous comment on metrics: I had forgotten about the outbound_http_errors_total metric, which is also incremented by these timeouts. The timeouts added in this PR will behave identically to ServiceProfile timeouts when incrementing this metric: we will increment the metric with the error="response timeout" label. This can be distinguished from failfast and load-shedding timeouts, which have their own labels. Similarly, we can distinguish these timeouts in the metrics from application 504s, since those do not increment the outbound_http_errors_total metric.

[mateiidavid]

Legible enough for me to review 🙃 looks good. The opaque route is confusing me a bit but think that's more due to the complexity of opaqueness, not because of this PR.