feat(tracing): add tracing support and configuration options for BBR by AtharvaPakade · Pull Request #2518 · kubernetes-sigs/gateway-api-inference-extension

AtharvaPakade · 2026-03-08T14:57:22Z

What type of PR is this?
/kind feature

What this PR does / why we need it:
This PR introduces BBR-specific tracing support by adding a dedicated pkg/bbr/tracing package.

The motivation behind a component-specific tracing package (rather than directly using the shared common tracing utilities) is to avoid conflicts in InstrumentationScope names between BBR and EPP. Each component should own its tracer with a distinct instrumentation scope name (e.g., gateway-api-inference-extension/bbr/extproc), ensuring traces from BBR and EPP don't overlap and remain distinguishable in the backend. Additionally, having a dedicated package per component allows us to add component-specific helpers that can enrich spans with attributes relevant to that server, keeping the tracing logic modular and tailored to each component's context.

On the configuration side, the body-based-routing Helm chart is updated to expose tracing options — enabling/disabling tracing, exporter endpoint, sampler, sampler arg, and service name — which get injected as OTEL environment variables into the BBR pod.

Currently, only BBR has been updated. EPP will follow the same pattern in a future PR.

Which issue(s) this PR fixes:
Fixes #2327

Does this PR introduce a user-facing change?:

BBR now supports OpenTelemetry tracing (opt-in). Enable it via the Helm
chart (`bbr.tracing.enabled: true`) or the `--tracing` flag, with options
to configure the exporter endpoint, sampler, and service name via envs.

- add tracing package for BBR-specific instrumentation and helpers - update `body-based-routing` helm chart to support configuring tracing via OTEL env vars (service name, exporter endpoint, sampler, sampler arg, and exporter type)

netlify · 2026-03-08T14:57:28Z

✅ Deploy Preview for gateway-api-inference-extension ready!

Name	Link
🔨 Latest commit	`bd60ece`
🔍 Latest deploy log	https://app.netlify.com/projects/gateway-api-inference-extension/deploys/69b03f01a153c0000852b49a
😎 Deploy Preview	https://deploy-preview-2518--gateway-api-inference-extension.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

k8s-ci-robot · 2026-03-08T14:57:32Z

Hi @AtharvaPakade. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work.

Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

nirrozenbaum · 2026-03-08T15:17:05Z

/ok-to-test

nirrozenbaum

@AtharvaPakade thanks!
this looks like a great start.
I left few comments.

@sallyom would be great if you can have a look as well :)

nirrozenbaum · 2026-03-08T15:20:27Z

+        {{- if not .Values.bbr.multiNamespace }}
        - name: NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        {{- end }}
+        {{- if .Values.bbr.tracing.enabled}}


out of curiosity, is leaving env with no values under it valid?
e.g., if multiNamespace is false and tracing.enabled is false.

It is valid and works, but it would be better to add another if to check if one of the two condition are true.

you can leave it as is for now since empty env is working.
there is a planned change for the multiNamespace anyway so we'll need to do some changes here.

nirrozenbaum · 2026-03-08T15:25:48Z

+	if opts.Tracing {
+		err := tracing.InitTracing(ctx, setupLog)
+		if err != nil {
+			setupLog.Error(err, "failed to initialize tracing")
+			return err
+		}
+	}
+


I understand that the intention is to keep separate traces between bbr/epp.
two points about this:

we should generalize tracing helper functions under common/observability/tracing in a consumable way, to allow both epp/bbr consume it and make their own traces independently.

at the same time, we should also allow (in epp, not here) to continue the same trace from bbr. technically speaking, if we want to see a full trace of the request (including bbr + epp) that is very useful.
having said that, this point is not required in current PR (can be done in follow ups), but just something to keep in mind.

I do plan to do that, but just avoided it for now in this PR as epp changes will also required. In futhure PR i will make this uniform, just to give a hint i was planning some pacakge level tracing config that will be initialized by common helper.

So i do have this in mind, i have not drafted a proposal yet but there are few things i have indetified which can add a lot of values to tracing in general.

nirrozenbaum · 2026-03-08T15:26:03Z

+	"sigs.k8s.io/gateway-api-inference-extension/pkg/bbr/tracing"
 	logutil "sigs.k8s.io/gateway-api-inference-extension/pkg/common/observability/logging"
 	"sigs.k8s.io/gateway-api-inference-extension/pkg/common/observability/profiling"
+


nit: remove extra space.

Suggested change

nirrozenbaum · 2026-03-08T15:27:25Z

+        {{- if .Values.bbr.tracing.enabled}}
+        - --tracing=true
+        {{- else}}
+        - --tracing=false
+        {{- end}}


is it possible to align with other flags using key/value in separate cmd-line flags? does it work?

Suggested change

{{- if .Values.bbr.tracing.enabled}}

- --tracing=true

{{- else}}

- --tracing=false

{{- end}}

{{- if .Values.bbr.tracing.enabled}}

- --tracing

- "true"

{{- else}}

- --tracing

- "false"

{{- end}}

This will not work because it resolves to --tracing <value>. Since tracing is a boolean flag, the Go flag parser treats the presence of --tracing as true, and the following value is interpreted as a positional argument rather than the flag’s value.

nirrozenbaum · 2026-03-08T15:28:22Z

        {{- end }}
+        {{- if .Values.bbr.tracing.enabled}}
+        - name: OTEL_SERVICE_NAME
+          value: {{ .Values.bbr.tracing.otelServiceName | default "gateway-api-inference-extension/bbr" }}


can we move default to values.yaml where we hold all other defaults?

nirrozenbaum · 2026-03-08T15:29:54Z

    v: 3

+  tracing:
+    enabled: true


can you align with epp and put default enabled=false?

Suggested change

enabled: true

enabled: false

nirrozenbaum · 2026-03-08T15:32:47Z

+func InitTracing(ctx context.Context, logger logr.Logger) error {
+	_, ok := os.LookupEnv("OTEL_SERVICE_NAME")
+	if !ok {
+		os.Setenv("OTEL_SERVICE_NAME", defaultServiceName)
+	}
+
+	return tracingutils.InitTracing(ctx, logger)
+}


can we instead pass in default service name and completely reuse in both places?

nirrozenbaum · 2026-03-08T15:33:57Z

+func GetExtProcTracer() trace.Tracer {
+	return otel.Tracer(
+		extprocInstrumentationName,
+		trace.WithInstrumentationVersion(version.BuildRef),
+		trace.WithInstrumentationAttributes(
+			attribute.String("commit-sha", version.CommitSHA),
+		),
+	)
+}


this looks different than what we have in epp.
not a tracing expert, but we should probably align on best practice and reuse the same in both.
BuildRef and CommitSHA seems very useful IMO.

Like i mentioned earlier i do plan to make it uniform with EPP in future PR. The intention for adding these 2 funcs GetExtProcTracer, InitTracing was to avoid same service names and instrumentation scope for both server as this will mixup traces.

If you suggest, for now i can use default name in case of epp and pass bbr specific names to the common function. By removing the tracing package for now from bbr.

I recommend to keep it aligned in both places.
so in both epp/bbr runner, you can do something like:

tracingServiceName := env.GetEnvString(envVar, "<THE-DEFAULT-PER-COMPONENT>", setupLog) tracing.InitTracing(... current args ..., additional-arg: tracingServingName)

so instead of diverging, and then try to align things - it would probably be better to align on current tracing from epp, then if things need to be added (e.g., like CommitSHA and buildRef which seems useful), we add it in the common packages, and then both epp/bbr get those changes.

… scope names - accept `defaultServiceName` as a parameter in `InitTracing` instead of hardcoding it, allowing BBR and EPP to set their own default service names - remove the BBR-specific tracing package; tracer initialization is now handled by the common tracing package - use distinct instrumentation scope names per component (`/bbr/extproc`, `/epp/extproc`) with version and commit-sha attributes - update Helm chart defaults: disable tracing by default and set explicit default service name for BBR

AtharvaPakade

@nirrozenbaum , I have made the discussed changes

nirrozenbaum · 2026-03-10T17:54:53Z

@sallyom @Gregory-Pereira @JeffLuoo can you have a look to make sure this is aligned with your view?

JeffLuoo · 2026-03-11T14:20:06Z

/lgtm

Thanks for the work!

nirrozenbaum

/approve

Thanks!

k8s-ci-robot · 2026-03-11T15:07:28Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: AtharvaPakade, nirrozenbaum

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [nirrozenbaum]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

…ubernetes-sigs#2518) * feat(tracing): add tracing support and configuration options for BBR - add tracing package for BBR-specific instrumentation and helpers - update `body-based-routing` helm chart to support configuring tracing via OTEL env vars (service name, exporter endpoint, sampler, sampler arg, and exporter type) * refactor(tracing): make InitTracing generic and unify instrumentation scope names - accept `defaultServiceName` as a parameter in `InitTracing` instead of hardcoding it, allowing BBR and EPP to set their own default service names - remove the BBR-specific tracing package; tracer initialization is now handled by the common tracing package - use distinct instrumentation scope names per component (`/bbr/extproc`, `/epp/extproc`) with version and commit-sha attributes - update Helm chart defaults: disable tracing by default and set explicit default service name for BBR

…ubernetes-sigs/gateway-api-inference-extension#2518) * feat(tracing): add tracing support and configuration options for BBR - add tracing package for BBR-specific instrumentation and helpers - update `body-based-routing` helm chart to support configuring tracing via OTEL env vars (service name, exporter endpoint, sampler, sampler arg, and exporter type) * refactor(tracing): make InitTracing generic and unify instrumentation scope names - accept `defaultServiceName` as a parameter in `InitTracing` instead of hardcoding it, allowing BBR and EPP to set their own default service names - remove the BBR-specific tracing package; tracer initialization is now handled by the common tracing package - use distinct instrumentation scope names per component (`/bbr/extproc`, `/epp/extproc`) with version and commit-sha attributes - update Helm chart defaults: disable tracing by default and set explicit default service name for BBR

k8s-ci-robot added the kind/feature Categorizes issue or PR as related to a new feature. label Mar 8, 2026

k8s-ci-robot requested review from elevran and liu-cong March 8, 2026 14:57

k8s-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Mar 8, 2026

k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Mar 8, 2026

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Mar 8, 2026

nirrozenbaum reviewed Mar 8, 2026

View reviewed changes

k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Mar 10, 2026

AtharvaPakade commented Mar 10, 2026

View reviewed changes

k8s-ci-robot assigned JeffLuoo Mar 11, 2026

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Mar 11, 2026

nirrozenbaum approved these changes Mar 11, 2026

View reviewed changes

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Mar 11, 2026

k8s-ci-robot merged commit a8d7948 into kubernetes-sigs:main Mar 11, 2026
11 checks passed

AtharvaPakade mentioned this pull request Mar 14, 2026

[tracing] Enhanced Tracing Configuration and Trace Quality #2581

Open

Uh oh!

Conversation

AtharvaPakade commented Mar 8, 2026

Uh oh!

netlify Bot commented Mar 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for gateway-api-inference-extension ready!

Uh oh!

k8s-ci-robot commented Mar 8, 2026

Uh oh!

nirrozenbaum commented Mar 8, 2026

Uh oh!

nirrozenbaum left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nirrozenbaum Mar 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nirrozenbaum Mar 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AtharvaPakade left a comment

Choose a reason for hiding this comment

Uh oh!

nirrozenbaum commented Mar 10, 2026

Uh oh!

JeffLuoo commented Mar 11, 2026

Uh oh!

nirrozenbaum left a comment

Choose a reason for hiding this comment

Uh oh!

k8s-ci-robot commented Mar 11, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

netlify Bot commented Mar 8, 2026 •

edited

Loading

nirrozenbaum Mar 8, 2026 •

edited

Loading

nirrozenbaum Mar 8, 2026 •

edited

Loading