feat(storage): Forward configured headers to Elasticsearch/OpenSearch backend#8544
Merged
yurishkuro merged 15 commits intoMay 10, 2026
Merged
Conversation
Add a new internal/headerforwarding package that extracts specified headers from inbound HTTP or gRPC requests and forwards them as gRPC metadata on outbound calls to the remote storage backend. Each forwarded header is configured with: - http_name: HTTP request header to extract - grpc_name: inbound gRPC metadata key (falls back to http_name) - header_role: semantic role (username|email), informational only - grpc_outbound_name: outbound metadata key (falls back to grpc_name/http_name) Wire into jaeger-query (HTTP middleware + gRPC server interceptors) and the gRPC storage factory (client interceptors). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: Yuri Shkuro <github@ysh.us>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: Yuri Shkuro <github@ysh.us>
…e values Matches http.Header.Get() semantics which also returns the first value. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: Yuri Shkuro <github@ysh.us>
…andler chain Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: Yuri Shkuro <github@ysh.us>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: Yuri Shkuro <github@ysh.us>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: Yuri Shkuro <github@ysh.us>
…to Elasticsearch requests Signed-off-by: ChaitanyaD48 <chaitanya.d48@gmail.com>
…sing headerforwarding RoundTripper Signed-off-by: ChaitanyaD48 <chaitanya.d48@gmail.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Adds end-to-end header forwarding support so identity headers captured by jaeger-query can be propagated to HTTP-based storage backends (Elasticsearch/OpenSearch), complementing the existing gRPC storage forwarding support.
Changes:
- Introduces an HTTP
RoundTripperininternal/headerforwardingthat injects captured headers fromcontext.Contextinto outbound HTTP requests. - Wraps Elasticsearch/OpenSearch HTTP transports (v7 olivere + v8 client) with the new forwarding
RoundTripper, ensuring forwarded headers are applied on outbound requests. - Adds/extends header-forwarding capture and propagation wiring in
jaeger-query(HTTP middleware + gRPC interceptors) and updates unit/integration tests.
Reviewed changes
Copilot reviewed 18 out of 18 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| internal/storage/v2/grpc/factory.go | Installs header-forwarding gRPC client interceptors when configured. |
| internal/storage/v2/grpc/factory_test.go | Adds factory test coverage for enabling header forwarding. |
| internal/storage/v2/grpc/config.go | Extends gRPC storage config with header_forwarding. |
| internal/storage/elasticsearch/config/config.go | Wraps ES transports with HTTP header-forwarding RoundTripper (v7 + v8). |
| internal/storage/elasticsearch/config/config_test.go | Adds httptest.Server coverage asserting forwarded headers reach outbound ES requests. |
| internal/headerforwarding/package_test.go | Adds goleak verification TestMain for the package tests. |
| internal/headerforwarding/http.go | Adds HTTP server middleware to capture configured headers into request context. |
| internal/headerforwarding/http_test.go | Adds tests for HTTP header capture middleware. |
| internal/headerforwarding/http_client.go | Adds HTTP client RoundTripper for propagating captured headers to outbound HTTP requests. |
| internal/headerforwarding/http_client_test.go | Adds unit tests for the forwarding RoundTripper. |
| internal/headerforwarding/grpc.go | Adds gRPC server/client interceptors for capture + propagation via metadata. |
| internal/headerforwarding/grpc_test.go | Adds unit tests for gRPC interceptors and fallback behaviors. |
| internal/headerforwarding/context.go | Adds context storage helpers for captured header pairs. |
| internal/headerforwarding/context_test.go | Adds tests for context round-tripping captured headers. |
| internal/headerforwarding/config.go | Adds ForwardedHeader config schema and name fallback helpers. |
| cmd/jaeger/internal/extension/jaegerquery/internal/server.go | Wires header capture into jaeger-query HTTP and gRPC server stacks. |
| cmd/jaeger/internal/extension/jaegerquery/internal/server_test.go | Adds tests validating capture into context for both HTTP and gRPC requests. |
| cmd/jaeger/internal/extension/jaegerquery/internal/flags.go | Adds header_forwarding config field to query options. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+16
to
+29
| // ForwardedHeader describes one header to be forwarded from inbound requests to outbound gRPC storage calls. | ||
| type ForwardedHeader struct { | ||
| // HTTPName is the name of the HTTP request header to extract on inbound HTTP requests. | ||
| HTTPName string `mapstructure:"http_name"` | ||
| // GRPCName is the name of the gRPC metadata key to extract on inbound gRPC requests. | ||
| // When empty, HTTPName is used as the fallback. | ||
| GRPCName string `mapstructure:"grpc_name"` | ||
| // Role describes the semantic meaning of the header value (e.g. username, email). | ||
| // Jaeger does not act on this today; it is informational for downstream consumers. | ||
| Role HeaderRole `mapstructure:"header_role"` | ||
| // GRPCOutboundName is the metadata key used when forwarding the value to the gRPC storage backend. | ||
| // When empty, GRPCName/HTTPName is used as the fallback (in that order). | ||
| GRPCOutboundName string `mapstructure:"grpc_outbound_name"` | ||
| } |
Comment on lines
+34
to
+39
| for _, c := range captured { | ||
| if c.Value == "" || c.Header == nil || c.Header.HTTPName == "" { | ||
| continue | ||
| } | ||
| cloned.Header.Set(c.Header.HTTPName, c.Value) | ||
| } |
Comment on lines
+32
to
+39
| // RoundTripper contract: do not mutate the inbound *http.Request. | ||
| cloned := req.Clone(req.Context()) | ||
| for _, c := range captured { | ||
| if c.Value == "" || c.Header == nil || c.Header.HTTPName == "" { | ||
| continue | ||
| } | ||
| cloned.Header.Set(c.Header.HTTPName, c.Value) | ||
| } |
Comment on lines
+23
to
+31
| func TestNewHTTPClientRoundTripper_NilBaseUsesDefault(t *testing.T) { | ||
| rt := headerforwarding.NewHTTPClientRoundTripper(nil) | ||
| require.NotNil(t, rt) | ||
| // We cannot call DefaultTransport here without making a real network call, | ||
| // but we can confirm the RoundTripper short-circuits when there are no | ||
| // captured headers and forwards to its base. Using a non-nil base verifies | ||
| // the wrapping behavior; the nil branch is exercised by construction above. | ||
| } | ||
|
|
|
|
||
| func appendOutgoing(ctx context.Context) context.Context { | ||
| for _, c := range CapturedFromContext(ctx) { | ||
| ctx = metadata.AppendToOutgoingContext(ctx, c.Header.outboundGRPCName(), c.Value) |
ChaitanyaD48
changed the title
ChaitanyaD48
changed the title
yurishkuro
reviewed
May 10, 2026
| if c.Value == "" || c.Header == nil || c.Header.HTTPName == "" { | ||
| continue | ||
| } | ||
| cloned.Header.Set(c.Header.HTTPName, c.Value) |
Member
There was a problem hiding this comment.
there must be a way to rewrite the header name
Contributor
Author
There was a problem hiding this comment.
Thanks for the review @yurishkuro! Fixed in the latest push. Mirrored the existing GRPCOutboundName pattern.
Comment on lines
+423
to
+426
| // Outermost wrapper: forward headers captured on the inbound request context | ||
| // (populated by the jaeger_query header_forwarding middleware/interceptors) | ||
| // onto every outbound request to Elasticsearch. | ||
| options.Transport = headerforwarding.NewHTTPClientRoundTripper(transport) |
Comment on lines
+24
to
+29
| rt := headerforwarding.NewHTTPClientRoundTripper(nil) | ||
| require.NotNil(t, rt) | ||
| // We cannot call DefaultTransport here without making a real network call, | ||
| // but we can confirm the RoundTripper short-circuits when there are no | ||
| // captured headers and forwards to its base. Using a non-nil base verifies | ||
| // the wrapping behavior; the nil branch is exercised by construction above. |
…client requests Signed-off-by: ChaitanyaD48 <chaitanya.d48@gmail.com>
…ack to HTTPName Signed-off-by: ChaitanyaD48 <chaitanya.d48@gmail.com>
…ack to HTTPName Signed-off-by: ChaitanyaD48 <chaitanya.d48@gmail.com>
Comment on lines
+40
to
+44
| if name == "" { | ||
| continue | ||
| } | ||
| cloned.Header.Set(name, c.Value) | ||
| } |
Comment on lines
+29
to
+31
| // HTTPOutboundName is the header name used when forwarding the value to an HTTP storage backend. | ||
| // When empty, HTTPName is used as the fallback. | ||
| HTTPOutboundName string `mapstructure:"http_outbound_name"` |
Comment on lines
+23
to
+30
| func TestNewHTTPClientRoundTripper_NilBaseUsesDefault(t *testing.T) { | ||
| rt := headerforwarding.NewHTTPClientRoundTripper(nil) | ||
| require.NotNil(t, rt) | ||
| // We cannot call DefaultTransport here without making a real network call, | ||
| // but we can confirm the RoundTripper short-circuits when there are no | ||
| // captured headers and forwards to its base. Using a non-nil base verifies | ||
| // the wrapping behavior; the nil branch is exercised by construction above. | ||
| } |
yurishkuro
approved these changes
May 10, 2026
yurishkuro
added
the
changelog:new-feature
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #8544 +/- ##
=======================================
Coverage 96.51% 96.51%
=======================================
Files 329 330 +1
Lines 17342 17364 +22
=======================================
+ Hits 16737 16759 +22
Misses 455 455
Partials 150 150
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
2 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which problem is this PR solving?
Resolves #8540
PR #8539 added a
header_forwardingconfig tojaeger_queryand wired it to the v2 gRPC storage backend.This PR adds the outbound HTTP forwarding hop so the same configuration works end-to-end for ElasticSearch/OpenSearch
This PR builds on top of #8539, which introduced the
internal/headerforwardingpackage along with the HTTP middleware and gRPC unary/stream server + client interceptors. It adds the missing HTTP-client analog and wires it into the Elasticsearch storage path.Description of the changes
internal/headerforwarding/http_client.goAdded a new
NewHTTPClientRoundTripper(base)implementation. It readsCapturedHeaders from the request context and injects them into outbound*http.Requests usingForwardedHeader.HTTPName.This acts as the HTTP equivalent of the existing
NewUnaryClientInterceptor/NewStreamClientInterceptor.internal/storage/elasticsearch/config/config.goWrapped the outbound HTTP transport with the new RoundTripper at both Elasticsearch client installation sites:
newElasticsearchV8(v8 client)getConfigOptions(v7 / olivere client)The wrapper is applied outside the existing TLS / bearer-token / body-fix transport chain so captured headers are appended last on outbound requests.
Header capture behavior remains unchanged.
Capture still occurs on the query-server side through
HTTPServerMiddlewareintroduced in feat(query): Add configurable header forwarding to gRPC storage backend #8539. This PR only adds the outbound HTTP forwarding hop for Elasticsearch / OpenSearch.How was this change tested?
Added unit tests in
internal/headerforwarding/http_client_test.gocovering:Added integration tests in
internal/storage/elasticsearch/config/config_test.gousing anhttptest.Serverto verify:Verified end-to-end locally against Elasticsearch 8 (
docker-compose/elasticsearch/v8) usingmitmwebas a reverse proxy.Configuration used:
Negative control — same curl without the headers; outbound ES requests do not carry X-Forwarded-*.
Checklist
make lint testAI Usage in this PR (choose one)
See AI Usage Policy.