Skip to content

Auth tokens leaked via otel_debug! header logging in OTLP tonic exporter #3367

@bryantbiggs

Description

@bryantbiggs

Related to #3021 and PR #3360.

PR #3360 does a good job redacting auth tokens from export error messages and interceptor error chains. But there's a separate code path it doesn't cover: the otel_debug! call in build_channel() that logs all headers in plaintext.

In opentelemetry-otlp/src/exporter/tonic/mod.rs, line 256:

otel_debug!(name: "TonicChannelBuilt", endpoint = endpoint_clone, timeout_in_millisecs = timeout.as_millis(), compression = format!("{:?}", compression), headers = format!("{:?}", headers_for_logging));

The headers_for_logging variable is a Vec<(String, String)> returned from parse_headers_from_env() (line 187, 415-435). It contains the raw key-value pairs from OTEL_EXPORTER_OTLP_HEADERS and signal-specific header env vars, with no redaction applied. If a user sets something like OTEL_EXPORTER_OTLP_HEADERS=authorization=Bearer my-secret-token, the full token ends up in debug output.

This is a real concern because:

  • Users often enable debug logging when troubleshooting connectivity issues, which is exactly when auth problems show up
  • Debug logs frequently end up in log aggregation systems, CI output, or support tickets
  • The variable is even named headers_for_logging, suggesting it was intended to be logged, but it was never sanitized

Suggested fix: redact header values for known sensitive header names (authorization, x-api-key, etc.) before logging, or only log header names without values. Something like:

let redacted: Vec<_> = headers_for_logging.iter()
    .map(|(k, v)| {
        let lower = k.to_lowercase();
        if lower == "authorization" || lower.contains("api-key") || lower.contains("token") {
            (k.clone(), "[REDACTED]".to_string())
        } else {
            (k.clone(), v.clone())
        }
    })
    .collect();
otel_debug!(name: "TonicChannelBuilt", ..., headers = format!("{:?}", redacted));

This could be addressed as a follow-up to PR #3360

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions