-
Notifications
You must be signed in to change notification settings - Fork 639
Description
Related to #3021 and PR #3360.
PR #3360 does a good job redacting auth tokens from export error messages and interceptor error chains. But there's a separate code path it doesn't cover: the otel_debug! call in build_channel() that logs all headers in plaintext.
In opentelemetry-otlp/src/exporter/tonic/mod.rs, line 256:
otel_debug!(name: "TonicChannelBuilt", endpoint = endpoint_clone, timeout_in_millisecs = timeout.as_millis(), compression = format!("{:?}", compression), headers = format!("{:?}", headers_for_logging));The headers_for_logging variable is a Vec<(String, String)> returned from parse_headers_from_env() (line 187, 415-435). It contains the raw key-value pairs from OTEL_EXPORTER_OTLP_HEADERS and signal-specific header env vars, with no redaction applied. If a user sets something like OTEL_EXPORTER_OTLP_HEADERS=authorization=Bearer my-secret-token, the full token ends up in debug output.
This is a real concern because:
- Users often enable debug logging when troubleshooting connectivity issues, which is exactly when auth problems show up
- Debug logs frequently end up in log aggregation systems, CI output, or support tickets
- The variable is even named
headers_for_logging, suggesting it was intended to be logged, but it was never sanitized
Suggested fix: redact header values for known sensitive header names (authorization, x-api-key, etc.) before logging, or only log header names without values. Something like:
let redacted: Vec<_> = headers_for_logging.iter()
.map(|(k, v)| {
let lower = k.to_lowercase();
if lower == "authorization" || lower.contains("api-key") || lower.contains("token") {
(k.clone(), "[REDACTED]".to_string())
} else {
(k.clone(), v.clone())
}
})
.collect();
otel_debug!(name: "TonicChannelBuilt", ..., headers = format!("{:?}", redacted));This could be addressed as a follow-up to PR #3360