Skip to content

fix: OTLP logging improvements #3343

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
cijothomas wants to merge 10 commits into
base: main
Choose a base branch
from
Open

fix: OTLP logging improvements #3343

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
97ebbaf
fix: OTLP logging improvmenets
cijothomas Feb 4, 2026
077b508
some logs
cijothomas Feb 4, 2026
e32c996
Merge branch 'main' into cijothomas/fixotlplogging
cijothomas Feb 4, 2026
3d9f3b1
better event name and http too
cijothomas Feb 4, 2026
55a6f20
nit tweak
cijothomas Feb 4, 2026
104d788
Merge branch 'main' into cijothomas/fixotlplogging
cijothomas Feb 5, 2026
03351c2
Apply suggestion from @utpilla
cijothomas Feb 10, 2026
79c6d7a
Apply suggestion from @utpilla
cijothomas Feb 10, 2026
ee681f4
Apply suggestion from @utpilla
cijothomas Feb 10, 2026
7b658dd
emit more when it is safe to do so
cijothomas Feb 10, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions opentelemetry-otlp/CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,13 @@

## vNext

- Improved error logging for gRPC and HTTP exporters. The status code is now
logged at WARN level, while potentially sensitive details (message, response
body) are logged at DEBUG level only. This prevents sensitive information
(like authentication tokens) from being logged at higher log levels.
**Note:** If you enable `DEBUG` level logging in production, these details
will still be visible. Use `WARN` or higher for production centralized logging.
[#3021](https://github.com/open-telemetry/opentelemetry-rust/issues/3021)
- Fix `NoHttpClient` error when multiple HTTP client features are enabled by using priority-based selection (`reqwest-client` > `hyper-client` > `reqwest-blocking-client`). [#2994](https://github.com/open-telemetry/opentelemetry-rust/issues/2994)
- Add partial success response handling for OTLP exporters (traces, metrics, logs) per OTLP spec. Exporters now log warnings when the server returns partial success responses with rejected items and error messages. [#865](https://github.com/open-telemetry/opentelemetry-rust/issues/865)
- Refactor `internal-logs` feature in `opentelemetry-otlp` to reduce unnecessary dependencies[3191](https://github.com/open-telemetry/opentelemetry-rust/pull/3192)
Expand Down
28 changes: 21 additions & 7 deletions opentelemetry-otlp/src/exporter/http/mod.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ use super::{
};
use crate::{ExportConfig, Protocol, OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_EXPORTER_OTLP_HEADERS};
use http::{HeaderName, HeaderValue, Uri};
use opentelemetry::otel_debug;
use opentelemetry::{otel_debug, otel_warn};
use opentelemetry_http::{Bytes, HttpClient};
use opentelemetry_proto::transform::common::tonic::ResourceAttributesWithSchema;
#[cfg(feature = "logs")]
Expand Down Expand Up @@ -488,7 +488,15 @@ impl OtlpHttpClient {

// Send request
let response = client.send_bytes(request).await.map_err(|e| {
HttpExportError::new(0, format!("Network error: {e:?}")) // Network error
// Connection errors (e.g., "Connection refused", DNS failures) typically
// indicate user-side misconfigurations and don't contain sensitive data,
// so it's safe to log the error message at WARN level.
otel_warn!(
name: "HttpClient.ExportFailed",
Copy link
Contributor

@utpilla utpilla Feb 5, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We are using the same event name HttpClient.ExportFailed for two different kinds of errors:

  • Network error from client.send_bytes
  • HTTP status error when response.status() is not successful.

Maybe we could use different event names for better diagnosability: HttpClient.NetworkError and HttpClientStatusError.

Copy link
Member Author

@cijothomas cijothomas Feb 5, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did consider that and ended up with same one. I am not 100% happy with this either.

I tried to distinguish between failures where we don't get a response at all and the ones where we get response, but it has status code indicating failure. But then realized that request client don't expose us a way to obtain StatusCode from its Error. Without having firmer control over the client, this is a decent tradeoff.
(The otel-http crate could offer additional logging, as it knows internals of the client it uses, but OTLP exporter does not know that much).

let me know if we should try to make this better or this is acceptable tradeoff.

url = request_uri.as_str(),
error = format!("{e}")
);
HttpExportError::new(0, "HTTP export failed".to_string())
})?;

let status_code = response.status().as_u16();
Expand All @@ -499,12 +507,18 @@ impl OtlpHttpClient {
.map(|s| s.to_string());

if !response.status().is_success() {
let message = format!(
"HTTP export failed. Url: {}, Status: {}, Response: {:?}",
request_uri,
status_code,
response.body()
otel_warn!(
name: "HttpClient.ExportFailed",
status_code = status_code,
url = request_uri.as_str()
);
// Response body may contain sensitive information,
// so log it at debug level only.
otel_debug!(
name: "HttpClient.ExportFailedDetails",
response_body = format!("{:?}", response.body())
);
let message = "HTTP export failed".to_string();
return Err(match retry_after {
Some(retry_after) => {
HttpExportError::with_retry_after(status_code, retry_after, message)
Expand Down
60 changes: 56 additions & 4 deletions opentelemetry-otlp/src/exporter/tonic/logs.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -94,8 +94,22 @@ impl LogExporter for TonicLogsClient {
.interceptor
.call(Request::new(()))
.map_err(|e| {
otel_warn!(
name: "TonicLogsClient.InterceptorFailed",
grpc_code = format!("{:?}", e.code())
);
// grpc_message and grpc_details may contain sensitive information,
// so log them at debug level only.
otel_debug!(
name: "TonicLogsClient.InterceptorFailedDetails",
grpc_message = e.message(),
grpc_details = format!("{:?}", e.details())
);
// Convert interceptor errors to tonic::Status for retry classification
tonic::Status::internal(format!("interceptor error: {e:?}"))
tonic::Status::internal(&format!(
"Logs export failed in interceptor with gRPC code: {:?}",
e.code()
))
})?
.into_parts();
Ok((inner.client.clone(), m, e))
Expand Down Expand Up @@ -137,9 +151,47 @@ impl LogExporter for TonicLogsClient {
.await
{
Ok(_) => Ok(()),
Err(tonic_status) => Err(OTelSdkError::InternalFailure(format!(
"export error: {tonic_status:?}"
))),
Err(tonic_status) => {
// For connection-related errors (Unavailable, Unknown, etc.), the message
// typically contains safe, actionable information (e.g., "Connection refused").
// For auth errors (Unauthenticated, PermissionDenied), the message may contain
// sensitive information, so we only log the code at WARN level.
let code = tonic_status.code();
let is_connection_error = matches!(
Copy link
Member

@lalitb lalitb Feb 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit - this ~35-40 lines of code seems to be repeated for all the signals - can have a shared helper method.

code,
tonic::Code::Unavailable
| tonic::Code::Unknown
| tonic::Code::DeadlineExceeded
| tonic::Code::ResourceExhausted
| tonic::Code::Aborted
| tonic::Code::Cancelled
);

if is_connection_error {
otel_warn!(
name: "TonicLogsClient.ExportFailed",
grpc_code = format!("{:?}", code),
grpc_message = tonic_status.message()
);
} else {
// For potentially sensitive errors (Unauthenticated, PermissionDenied, etc.),
// only log the code at WARN level.
otel_warn!(
name: "TonicLogsClient.ExportFailed",
grpc_code = format!("{:?}", code)
);
// Log message and details at debug level for sensitive error types.
otel_debug!(
name: "TonicLogsClient.ExportFailedDetails",
grpc_message = tonic_status.message(),
grpc_details = format!("{:?}", tonic_status.details())
);
}
Err(OTelSdkError::InternalFailure(format!(
"Logs export failed with gRPC code: {:?}",
code
)))
}
}
}

Expand Down
58 changes: 52 additions & 6 deletions opentelemetry-otlp/src/exporter/tonic/metrics.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -85,9 +85,18 @@ impl MetricsClient for TonicMetricsClient {
.interceptor
.call(Request::new(()))
.map_err(|e| {
tonic::Status::internal(format!(
"unexpected status while exporting {e:?}"
))
otel_warn!(
name: "TonicMetricsClient.InterceptorFailed",
grpc_code = format!("{:?}", e.code())
);
// grpc_message and grpc_details may contain sensitive information,
// so log them at debug level only.
otel_debug!(
name: "TonicMetricsClient.InterceptorFailedDetails",
grpc_message = e.message(),
grpc_details = format!("{:?}", e.details())
);
tonic::Status::internal("Metrics export failed in interceptor")
})?
.into_parts();
Ok((inner.client.clone(), m, e))
Expand Down Expand Up @@ -127,9 +136,46 @@ impl MetricsClient for TonicMetricsClient {
.await
{
Ok(_) => Ok(()),
Err(tonic_status) => Err(OTelSdkError::InternalFailure(format!(
"export error: {tonic_status:?}"
))),
Err(tonic_status) => {
// For connection-related errors (Unavailable, Unknown, etc.), the message
// typically contains safe, actionable information (e.g., "Connection refused").
// For auth errors (Unauthenticated, PermissionDenied), the message may contain
// sensitive information, so we only log the code at WARN level.
let code = tonic_status.code();
let is_connection_error = matches!(
code,
tonic::Code::Unavailable
| tonic::Code::Unknown
| tonic::Code::DeadlineExceeded
| tonic::Code::ResourceExhausted
| tonic::Code::Aborted
| tonic::Code::Cancelled
);

if is_connection_error {
otel_warn!(
name: "TonicMetricsClient.ExportFailed",
grpc_code = format!("{:?}", code),
grpc_message = tonic_status.message()
);
} else {
// For potentially sensitive errors (Unauthenticated, PermissionDenied, etc.),
// only log the code at WARN level.
otel_warn!(
name: "TonicMetricsClient.ExportFailed",
grpc_code = format!("{:?}", code)
);
// Log message and details at debug level for sensitive error types.
otel_debug!(
name: "TonicMetricsClient.ExportFailedDetails",
grpc_message = tonic_status.message(),
grpc_details = format!("{:?}", tonic_status.details())
);
}
Err(OTelSdkError::InternalFailure(
"Metrics export failed".into(),
Copy link
Member

@lalitb lalitb Feb 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

grpc_code should also be added here?

))
}
}
}

Expand Down
54 changes: 50 additions & 4 deletions opentelemetry-otlp/src/exporter/tonic/trace.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -96,8 +96,19 @@ impl SpanExporter for TonicTracesClient {
.interceptor
.call(Request::new(()))
.map_err(|e| {
otel_warn!(
name: "TonicTracesClient.InterceptorFailed",
grpc_code = format!("{:?}", e.code())
);
// grpc_message and grpc_details may contain sensitive information,
// so log them at debug level only.
otel_debug!(
name: "TonicTracesClient.InterceptorFailedDetails",
grpc_message = e.message(),
grpc_details = format!("{:?}", e.details())
);
// Convert interceptor errors to tonic::Status for retry classification
tonic::Status::internal(format!("interceptor error: {e:?}"))
tonic::Status::internal("Traces export failed in interceptor")
Copy link
Member

@lalitb lalitb Feb 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

grpc_code here too.

})?
.into_parts();
Ok((inner.client.clone(), m, e))
Expand Down Expand Up @@ -140,9 +151,44 @@ impl SpanExporter for TonicTracesClient {
.await
{
Ok(_) => Ok(()),
Err(tonic_status) => Err(OTelSdkError::InternalFailure(format!(
"export error: {tonic_status:?}"
))),
Err(tonic_status) => {
// For connection-related errors (Unavailable, Unknown, etc.), the message
// typically contains safe, actionable information (e.g., "Connection refused").
// For auth errors (Unauthenticated, PermissionDenied), the message may contain
// sensitive information, so we only log the code at WARN level.
let code = tonic_status.code();
let is_connection_error = matches!(
code,
tonic::Code::Unavailable
| tonic::Code::Unknown
| tonic::Code::DeadlineExceeded
| tonic::Code::ResourceExhausted
| tonic::Code::Aborted
| tonic::Code::Cancelled
);

if is_connection_error {
otel_warn!(
name: "TonicTracesClient.ExportFailed",
grpc_code = format!("{:?}", code),
grpc_message = tonic_status.message()
);
} else {
// For potentially sensitive errors (Unauthenticated, PermissionDenied, etc.),
// only log the code at WARN level.
otel_warn!(
name: "TonicTracesClient.ExportFailed",
grpc_code = format!("{:?}", code)
);
// Log message and details at debug level for sensitive error types.
otel_debug!(
name: "TonicTracesClient.ExportFailedDetails",
grpc_message = tonic_status.message(),
grpc_details = format!("{:?}", tonic_status.details())
);
}
Err(OTelSdkError::InternalFailure("Traces export failed".into()))
}
}
}

Expand Down
Loading