Make log correlation more configurable (support analysis services, e.g. Google, New Relic)

[matrixbot]

This issue has been migrated from #14413.

---

Description:

Thanks everyone who works on Synapse, it's really great.

The recent PR matrix-org/synapse#13801 added the ability to specify the header from which to pull a request ID. This is a great step toward log correlation, but it more-or-less requires manual action to correlate.

I am using Google's Cloud Logging (aka Stackdriver) and logs may be correlated automatically in their tooling if the structured logs contain a trace key with a value following a particular convention. Something similar exists for New Relic, see discussion at Seldaek/monolog#1735).

It would be awesome if Synapse could support this. I think it could be as simple as:

• Make the name of the request ID key, currently fixed to request, configurable. (E.g., I would set this to trace).
• Drop the auto-prefixing of the request with the method name. The method is already included in the log (method).

In my case, I am currently setting the trace-id with a little bit of lua magic with an openresty reverse proxy, e.g.:

server {
    location @default {
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $remote_addr;
        proxy_set_header X-Forwarded-Proto 'https';
        set_by_lua $trace_id 'return string.sub(ngx.req.get_headers()["traceparent"], 4, 35)';
        proxy_set_header trace-id $trace_id;
        set $upstream {{ .Env.UPSTREAM }};
        proxy_pass http://$upstream:{{ default .Env.UPSTREAM_PORT "8008" }};
    }
}

And if this were supported, I would simply update this to include the additional data Google is looking for, e.g. the project ID. Synapse would then be responsible for hairpinning that back to me.

Alternatively, I suppose this could be viewed as more an extension of the opentracing implementation, and Synapse could read out the Trace ID from the traceparent header and include it somehow in the logs. However, that wouldn't address the need for making the log value configurable (e.g., prefixing it with additional cloud provider namespacing.)