Set otel exporter headers value during runtime.

[huliang-microsoft]

Problem Statement

Sorry if this is not a feasible question.
I am trying to use otel exporter lib to export otel data to an otel collector. And I need to set auth header. This is what I do:

	kv := make(map[string]string)
	kv["Authorization"] = "Bearer " + token

	exporter := otlpmetrichttp.New(ctx, otlpmetrichttp.WithHeaders(kv))

The problem is that I cannot find an API to update the header once the exporter is initialized but when the token is expired, seems the only way is to create another exporter instance.

Is there a way to update the header in the exporter? or if there is not, is it Okay to add an API to update that?

[XSAM]

if there is not, is it Okay to add an API to update that?

I feel this is a legitimate request.

[dmathieu]

I see two ways we could do this:

• Add a WithHeadersFn method, which would be executed for every HTTP request, and retrieve the proper headers. It's up to the users to decide how those headers are updated.
• Add an UpdateHeaders method, which allows updating the headers into the exporter (either with a replace all, or with a merge of the existing ones, to only override the changed headers).

I'd lean towards the second option, as with the first one, the added method would be in a rather hot path. And the second one should allow more flexibility.

[pellared]

I am not sure if adding options like WithHeadersFn/UpdateHeaders would be good enough. Wouldn't the user also need to add a HTTP/gRPC response handler to invoke e.g. UpdateHeaders when the OTLP receiver/ingest returns e.g. 401 HTTP Status Code?

The client may know the token's expiration date. However, the token could be invalidated sooner than that.

[huliang-microsoft]

@dmathieu @XSAM @pellared Thanks for looking into this issue. While before this is implemented, I would like to find a workaround to achieve what I need.

I tried to create a new metric.MeterProvider and call otel.SetMeterProvider(meterProvider) during runtime, inside the new MeterProvider, I have a new metricExporter with the updated token. But it seems like the new MeterProvider is only added but not replacing the old one. Is there a way to replace the existing MeterProvider registered so I can use the new MeterProvider with the new token? Or is there other way to do that?

Thanks for the help!

[dmathieu]

Providers are additive, not replacing each other. So this approach won't be valid.
Also, replacing the meter provider would reset cumulative metrics, which is probably not what you want.

I would instead setup a custom exporter, with a way to replace the header value at runtime.
Then, that exporter can hold an OTLP exporter, and replace it with a new one every time the header changes.

[shalinibani]

I am also looking for this feature to update auth headers dynamically.

[shalinibani]

I'm not sure why my previous comment was marked as spam. I was genuinely interested in this feature and wanted to give this request more visibility, as I’m experiencing the same issue as @huliang-microsoft in my day to day work.

[dmathieu]

There's already a draft PR from @kevingentile
#6362

However, this looks like it may be picked up on the specification side: open-telemetry/opentelemetry-specification#4390
Rather than introduce a public API that may not be compliant with the way the spec ends up being, I'd rather not rush this.

If you wish to help move this forward, you can join in on the spec discussion.

[kevingentile]

The proposed changes in #6362 now include dynamic header support for logs, metrics, and trace. I recognize that the community may desire a more elegant solution, but the proposed changes are only a small extension of the public API and can be used to enable OAuth2 support.

[psx95]

The topic being discussed in open-telemetry/opentelemetry-specification#4390 seems to be targeted toward a very specific use-case of supporting authentication in OTLP exporters. It's likely gonna introduce the concept of 'authenticators'.

While the initial motivation behind supporting dynamic headers was supporting auth token refreshes, dynamic headers seem like a more general-purpose approach that could apply to other use-cases and I am wondering if the auth specific approach being discussed as a spec issue can co-exist with something like #6362.