[tracing] Enhanced Tracing Configuration and Trace Quality

[AtharvaPakade]

Summary

This issue proposes improvements to tracing configuration and trace quality for the Gateway API Inference Extension, building upon the discussion in PR #2518 and issue #2327. The focus is on making tracing more configurable, performant, and insightful for end-to-end request tracing across all components.

Background

The project currently uses environment variables for tracing configuration through the InitTracing function in pkg/common/observability/tracing/telemetry.go . EPP/BBR already implements tracing with a root span named "gateway.request" , and BBR is moving toward pluggable architecture that would benefit from enhanced tracing.

Proposed Improvements

1. Service-Level Configuration Structure

Problem: Current configuration relies on environment variables, which is less user-friendly and harder to validate.

Solution: Create a dedicated configuration struct for tracing settings that can be:

• Passed through the application stack
• Validated at startup
• Easily extended with new options
• Integrated with existing Helm configuration

2. Conditional Tracing in ExtProc Server

Problem: As we add richer tracing (events, sub-spans), the overhead may become significant for high-throughput scenarios.

Solution: Pass tracing configuration to the extproc server to:

• Conditionally enable/disable tracing at runtime
• Allow per-request tracing decisions based on sampling or headers
• Reduce overhead when detailed tracing isn't needed

Open Questions:

• Does the conditional check itself introduce overhead that offsets the benefits?
• What is the acceptable overhead threshold for production workloads?

3. End-to-End Trace Context Propagation

Problem: No unified view of request latency across all components in the request path.

Solution: Implement W3C trace context propagation across:

Client → Gateway → BBR → Gateway → EPP → Model

This would enable:

• Component-level latency analysis
• Bottleneck identification
• Request correlation across services
• Performance optimization insights

4. Richer Plugin-Scoped Sub-spans

Problem: Current tracing doesn't provide detailed insights into plugin execution.

Solution: Automatically create sub-spans for :

• Each BBR plugin execution
• Each EPP scheduling plugin
• Plugin configuration attributes (maybe having a plugin level conf)
• Plugin-specific metrics and events

Open Questions:

• Is the overhead of automatic sub-spans worth the observability benefit?
• Should we provide root spans to plugins by default and let plugins decide on sub-span creation, rather than enforcing out-of-box instrumentation?

Discussion & Feedback Needed

looking for community discussion on:

1. Whether to proceed: Should we implement all four improvements together, or prioritize specific areas? Are there points that don't align with the project's direction?

2. Feedback on value: Do these improvements address real pain points? Is the observability benefit worth the added complexity?

3. Specific considerations while building:

   • Acceptable overhead threshold for conditional tracing checks
   • Plugin instrumentation philosophy (automatic vs. opt-in)
   • Shared vs. separate tracing configs for BBR and EPP
   • Backward compatibility strategy for environment variables
   • Should end-to-end propagation be separate or bundled?

4. General suggestions:

   • Alternative approaches to reduce tracing overhead
   • Lessons learned from similar pluggable observability systems
   • Any architectural concerns we should consider

cc @nirrozenbaum @sallyom @JeffLuoo - please share your thoughts on the proposed improvements and open questions above. Feel free to tag anyone else who can add valuable insights to this discussion.

[nirrozenbaum]

@AtharvaPakade thanks for creating this issue.
as I already told few times, I'm not a tracing expert 😃, but I'll add my two cents:

Service-Level Configuration Structure

this sounds like a good idea. the usage of env vars is not convenient when the number of vars starts to grow.
having a structured configuration + support through helm sounds like the right way to go.

Conditional Tracing in ExtProc Server

I think this is another good idea. we can avoid conditionals if we create a NoOpTracer when the tracing is disabled in the configuration. then we should have no conditionals across the code, which also keeps the code cleaner.
to keep it simple I would start with enabling per-request traces by a dedicated header. in a later stage if we want to add sampling, we can inject the header based on the sample.

End-to-End Trace Context Propagation

just to be accurate here, the flow is:

Client → Gateway → BBR → Gateway → EPP → Gateway → Model Server

Sounds to me like for doing this we need to allow injection of trace.
I think it has a lot of value being able to have a single trace for the full flow.
This may also include P/D disaggregation use cases, where instead of having a single Model Server, the request might be sent to prefill worker and then to decode worker (P/D is currently done in llm-d).
having said the above, this sounds like an advanced scenario. I would probably keep it to "phase 2".

Richer Plugin-Scoped Sub-spans

I think the points you wrote have value. I have no good answer about when it's right to have span and when sub-span but I assume our tracing experts have more insights on this.

in terms of implementation plan, if you ask me I would start with a phased approach, taking each of the points in your list as a phase, exactly in the order you wrote them.
I would try to align tracing config for both components, but wouldn't force it.
if it fits naturally that's great. if not it's also valid to create different configs.

I would wait for feedback from @sallyom or @JeffLuoo before making any progress.

[JeffLuoo]

Thank you for the proposal, here is my thought:

Current configuration relies on environment variables, which is less user-friendly and harder to validate.

Environment variables remain a standard (and stable) way to configure OTel SDK: https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/, but declarative config became stable as well: https://opentelemetry.io/blog/2025/declarative-config/#go. The declarative config gives you a structured way to configure the OTel SDK (e.g., a configMap) compared to environment variables.

Conditional Tracing in ExtProc Server

We can consider strategy https://opentelemetry.io/blog/2022/tail-sampling/ where we provide guidance on which trace span to keep / drop.

Does the conditional check itself introduce overhead that offsets the benefits?

As @sallyom also mentioned, the observability sig needs to work with Benchmark group to run benchmarking with trace enabled.

Whether to proceed: Should we implement all four improvements together, or prioritize specific areas? Are there points that don't align with the project's direction?

IMO, we should prioritize:

1. More detailed spans from plugins and components. Context propagation should be handled when we implement it.

2. Sampling strategy.

3. Declarative config support.

Feedback on value: Do these improvements address real pain points? Is the observability benefit worth the added complexity?

Given IGW is now merging with llm-d. Observability work should prioritize the top use cases that llm-d supports (P/D, MoE?). @nirrozenbaum can provide some insights here. We need the community to provide feedbacks on the pain points that observability can help resolve.

[AtharvaPakade]

Thanks @nirrozenbaum and @JeffLuoo for your suggestions — I've put together an early PoC + design doc capturing the ideas and have a few open questions around tail sampling, span granularity and streaming. Would love everyone's thoughts here or in the doc: link

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale