Skip to content

fix(otlp): prevent auth tokens from leaking in export error messages#3360

Open
bryantbiggs wants to merge 1 commit intoopen-telemetry:mainfrom
bryantbiggs:fix/redact-auth-tokens-in-error-messages
Open

fix(otlp): prevent auth tokens from leaking in export error messages#3360
bryantbiggs wants to merge 1 commit intoopen-telemetry:mainfrom
bryantbiggs:fix/redact-auth-tokens-in-error-messages

Conversation

@bryantbiggs
Copy link
Contributor

@bryantbiggs bryantbiggs commented Feb 20, 2026

Summary

Fixes #3021
Supersedes #3108
Supersedes #3343

When gRPC/HTTP errors occur, tonic::Status was Debug-formatted into error messages, leaking Bearer tokens echoed back by the server. The HTTP exporter similarly included full response bodies in error messages.

Changes

gRPC exporters — Two macro_rules! macros (handle_tonic_export_error!, handle_interceptor_error!) defined once in tonic/mod.rs and used by all three signal exporters (traces, metrics, logs):

  • Connection errors (Unavailable, DeadlineExceeded, ResourceExhausted, Aborted, Cancelled): gRPC code + message at DEBUG
  • Potentially sensitive errors (Unknown, Unauthenticated, PermissionDenied, etc.): gRPC code, message, and details at DEBUG only
  • Interceptor errors: always treated as potentially sensitive (DEBUG only)
  • OTelSdkError returned to callers contains only the gRPC code, never the message
  • Signal-specific event names via concat!() (e.g., TonicLogsClient.ExportFailed)
  • No WARN logging in exporters — SDK processors already log returned errors via otel_error!

HTTP exporter — DEBUG-only logging:

  • Network errors: DEBUG with HttpClient.NetworkError
  • Status errors: status code + response body at DEBUG (HttpClient.StatusError)
  • No WARN logging in exporters — SDK processors already log returned errors via otel_error!

Review feedback addressed

From #3343:

  • Shared macros to eliminate duplicated error handling across signal exporters (@lalitb)
  • gRPC code included in all InternalFailure and interceptor error messages (@utpilla, @lalitb)
  • Distinct HTTP event names for network vs status errors (@utpilla)
  • DEBUG caveat in CHANGELOG (@utpilla)

From #3360:

  • Unknown moved to potentially sensitive errors — too broad, could carry sensitive server responses (@lalitb)
  • Removed WARN logging from exporters to avoid duplicate log entries with SDK processor otel_error! (@lalitb)

Test plan

  • 5 unit tests for error sanitization macros (token redaction, exporter name, gRPC codes)
  • cargo clippy --all-features -- -D warnings passes
  • cargo test --all-features — 111 tests pass
  • cargo fmt -- --check passes

@bryantbiggs bryantbiggs requested a review from a team as a code owner February 20, 2026 02:58
@bryantbiggs bryantbiggs force-pushed the fix/redact-auth-tokens-in-error-messages branch from 2470648 to f92b4b5 Compare February 20, 2026 02:59
@codecov
Copy link

codecov bot commented Feb 20, 2026

Codecov Report

❌ Patch coverage is 72.97297% with 20 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.3%. Comparing base (3c41f29) to head (144d6f0).
⚠️ Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
opentelemetry-otlp/src/exporter/http/mod.rs 0.0% 8 Missing ⚠️
opentelemetry-otlp/src/exporter/tonic/logs.rs 0.0% 3 Missing ⚠️
opentelemetry-otlp/src/exporter/tonic/metrics.rs 0.0% 3 Missing ⚠️
opentelemetry-otlp/src/exporter/tonic/mod.rs 94.7% 3 Missing ⚠️
opentelemetry-otlp/src/exporter/tonic/trace.rs 0.0% 3 Missing ⚠️
Additional details and impacted files
@@          Coverage Diff          @@
##            main   #3360   +/-   ##
=====================================
  Coverage   82.2%   82.3%           
=====================================
  Files        128     128           
  Lines      24626   24685   +59     
=====================================
+ Hits       20266   20317   +51     
- Misses      4360    4368    +8     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@bryantbiggs bryantbiggs force-pushed the fix/redact-auth-tokens-in-error-messages branch 2 times, most recently from 0023b45 to 1732e8a Compare February 20, 2026 03:15
// Connection errors (e.g., "Connection refused", DNS failures) typically
// indicate user-side misconfigurations and don't contain sensitive data,
// so it's safe to log the error message at WARN level.
otel_warn!(
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We emit otel_warn! here on export failure, but the SDK processors (BatchLogProcessor, BatchSpanProcessor, PeriodicReader, etc.) already log the returned error via otel_error!. This would result in duplicate log entries for the same export failure. Should we avoid logging in the macros and rely on the existing SDK-level logging instead?

let is_connection_error = matches!(
code,
tonic::Code::Unavailable
| tonic::Code::Unknown
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think Unknown is pretty broad and could carry sensitive server responses. Might be safer to treat it as potentially sensitive.

@bryantbiggs bryantbiggs force-pushed the fix/redact-auth-tokens-in-error-messages branch from 1732e8a to 144d6f0 Compare February 20, 2026 13:25
Copy link
Contributor Author

@bryantbiggs bryantbiggs left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed both review comments:

  1. Duplicate logging (http/mod.rs): Downgraded all exporter-level logging from otel_warn! to otel_debug! in both the HTTP exporter and the tonic macros (handle_tonic_export_error!, handle_interceptor_error!). SDK processors will handle the WARN/ERROR level logging for returned errors.

  2. Unknown too broad (tonic/mod.rs): Moved Unknown out of the is_connection_error match. It's now treated as potentially sensitive alongside Unauthenticated, PermissionDenied, etc.

Copy link
Member

@lalitb lalitb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice work!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: OTEL logs raw Authentication token to std-out on auth error

2 participants