Skip to content

Commit 3db9b65

Browse files
kariyclaude
andauthored
fix(tracing): install W3C TraceContext propagator in OTLP path (#532)
## Summary The OTLP init path never installed a text-map propagator, so katana silently dropped inbound `traceparent` headers under that backend — every request became a fresh root trace even when the caller had a live trace context. The `gcloud` path installs a `GoogleTraceContextPropagator`; OTLP was missing the equivalent W3C one. One-line fix: add `opentelemetry::global::set_text_map_propagator(TraceContextPropagator::new())` in `otlp::init_tracer`. ## Why this matters `crates/rpc/rpc-server/src/middleware/...` wraps incoming HTTP in a `tower_http::TraceLayer` configured with `GoogleStackDriverMakeSpan`, which calls `opentelemetry::global::get_text_map_propagator(|p| p.extract(&HeaderExtractor(req.headers())))`. With no propagator installed, `get_text_map_propagator` returns a `NoopTextMapPropagator` and the extract call silently produces an empty context. Result: `span.set_parent(cx)` is effectively a no-op, and the exported span starts a new trace instead of chaining to the caller's. ## Repro (before this PR) ```bash # Start jaeger v2 as an OTLP collector: jaeger --config file:/tmp/jaeger-config.yaml # OTLP gRPC on :4317, UI on :16686 # Start katana with OTLP: katana --tracer.otlp --tracer.otlp-endpoint http://localhost:4317 --http.port 5055 # Send a request with a synthetic traceparent: curl http://localhost:5055 \ -H 'traceparent: 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01' \ -d '{"jsonrpc":"2.0","method":"starknet_chainId","id":1}' # Check jaeger — the exported trace has a random trace_id, NOT 0af7651916cd43dd8448eb211c80319c curl "http://localhost:16686/api/traces?service=katana&limit=1" ``` After this PR, the exported `http_request` span has `trace_id=0af7651916cd43dd8448eb211c80319c` — the caller's context is preserved, and any spans katana fans out downstream chain under the same trace. ## Discovered during Wiring distributed tracing into the cartridge sidecar services ([cartridge-gg/vrf#46](cartridge-gg/vrf#46), [cartridge-gg/paymaster#15](cartridge-gg/paymaster#15)). Those PRs correctly install the W3C propagator; this PR closes the gap on the katana side so the three-service chain stitches under one trace_id in the collector. --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1 parent 3e40686 commit 3db9b65

File tree

3 files changed

+83
-2
lines changed

3 files changed

+83
-2
lines changed

crates/tracing/src/otlp.rs

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
use anyhow::Result;
22
use opentelemetry::trace::TracerProvider;
33
use opentelemetry_otlp::SpanExporterBuilder;
4+
use opentelemetry_sdk::propagation::TraceContextPropagator;
45
use opentelemetry_sdk::trace::{RandomIdGenerator, SdkTracerProvider};
56
use opentelemetry_sdk::Resource;
67

@@ -31,6 +32,15 @@ pub fn init_tracer(otlp_config: &OtlpConfig) -> Result<opentelemetry_sdk::trace:
3132
.with_resource(resource)
3233
.build();
3334

35+
// Install a W3C TraceContext propagator so the `MakeSpan` used by the
36+
// RPC server's `tower_http::TraceLayer` can extract inbound `traceparent`
37+
// headers and chain exported spans under the caller's trace_id. Without
38+
// this, every inbound request starts a fresh root trace even when the
39+
// caller sent a trace context — breaking distributed tracing across
40+
// services. The `gcloud` path installs its own propagator; the OTLP
41+
// path was missing one.
42+
opentelemetry::global::set_text_map_propagator(TraceContextPropagator::new());
43+
3444
opentelemetry::global::set_tracer_provider(provider.clone());
3545

3646
Ok(provider.tracer("katana"))

docs/cartridge.md

Lines changed: 71 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -428,6 +428,77 @@ Sidecar versions are pinned in `sidecar-versions.toml` at the repository root an
428428

429429
> Source: `crates/cli/src/sidecar/mod.rs`
430430
431+
## Distributed tracing
432+
433+
Katana and both sidecars (`paymaster-service`, `vrf-server`) emit OpenTelemetry spans via OTLP, so a single request that fans out from Katana through the sidecars produces one end-to-end trace in the collector.
434+
435+
### Wire protocol
436+
437+
| Service | Transport | Default endpoint |
438+
|---------|-----------|------------------|
439+
| `katana` | OTLP **gRPC** | `http://localhost:4317` |
440+
| `vrf-server` | OTLP **gRPC** | `http://localhost:4317` |
441+
| `paymaster-service` | OTLP **HTTP** | `http://localhost:4318/v1/traces` |
442+
443+
Context is propagated using the W3C **TraceContext** format (`traceparent` header). Any OTLP-compatible collector that accepts both gRPC and HTTP on the standard ports (Jaeger v2, `otel-collector`, Tempo, etc.) will stitch the three services under one `trace_id`.
444+
445+
### Enabling it
446+
447+
**Katana:**
448+
```bash
449+
katana --tracer.otlp --tracer.otlp-endpoint http://localhost:4317 ...
450+
```
451+
452+
See [`--tracer.gcloud`](../README.md) for the Google Cloud Trace alternative, which uses the `X-Cloud-Trace-Context` propagator instead of W3C.
453+
454+
**vrf-server** (when running standalone or via `--vrf.bin`):
455+
```bash
456+
vrf-server --tracer.otlp --tracer.otlp-endpoint http://localhost:4317 ...
457+
```
458+
459+
**paymaster-service** (configured via the profile JSON that katana writes in sidecar mode):
460+
```json
461+
{
462+
"prometheus": {
463+
"endpoint": "http://localhost:4318",
464+
"token": null
465+
}
466+
}
467+
```
468+
469+
> The profile field is named `prometheus` for historical reasons but actually configures OTLP trace and metric export. Setting it to `null` disables telemetry.
470+
471+
### What gets traced
472+
473+
| Service | Root span | Child spans |
474+
|---------|-----------|-------------|
475+
| `katana` | `http_request` (tower-http) | `rpc_call`, `db_get`, `db_put`, `db_txn_ro_create`, stage/pipeline spans |
476+
| `paymaster-service` | `http_request` (tower-http) | `paymaster_health`, `paymaster_buildTransaction`, `paymaster_executeTransaction`, `paymaster_executeRawTransaction`, ... (from `#[instrument]` on each RPC method) |
477+
| `vrf-server` | `http_request` (tower-http) | handler-level spans when `#[instrument]` is applied (not wired by default) |
478+
479+
Inbound `traceparent` headers are extracted by a `tower_http::TraceLayer` with a custom `MakeSpan` that calls the globally installed text-map propagator. Outbound HTTP calls between services (Katana -> paymaster, CartridgeApi -> vrf-server) do not yet inject `traceparent` automatically — callers that want full chain visibility must set the header themselves, or the service will start a fresh root span.
480+
481+
### End-to-end example
482+
483+
```bash
484+
# Start an OTLP collector (Jaeger v2 shown here — UI on :16686):
485+
jaeger --config file:/path/to/jaeger-config.yaml
486+
487+
# Start katana with OTLP tracing:
488+
katana --chain-id SN_SEPOLIA \
489+
--paymaster --cartridge.paymaster --vrf \
490+
--tracer.otlp --tracer.otlp-endpoint http://localhost:4317
491+
492+
# Send a request with a synthesized W3C trace context:
493+
curl http://localhost:5050 \
494+
-H 'traceparent: 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01' \
495+
-d '{"jsonrpc":"2.0","method":"cartridge_addExecuteOutsideTransaction","params":[...],"id":1}'
496+
497+
# All three services emit spans under trace_id 0af7651916cd43dd8448eb211c80319c.
498+
```
499+
500+
> Source: `crates/tracing/src/otlp.rs`, `crates/tracing/src/gcloud.rs`; [cartridge-gg/vrf#46](https://github.com/cartridge-gg/vrf/pull/46), [cartridge-gg/paymaster#15](https://github.com/cartridge-gg/paymaster/pull/15)
501+
431502
## Error codes
432503

433504
| Code | Variant | Message |

sidecar-versions.toml

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,9 @@
11
[paymaster-service]
22
repo = "https://github.com/cartridge-gg/paymaster"
3-
rev = "4748365"
3+
rev = "8fc62f26f88cbe150b28104faa158af9188606ab"
44
package = "paymaster-service"
55

66
[vrf-server]
77
repo = "https://github.com/cartridge-gg/vrf.git"
8-
rev = "6d1c0f60a53558f19618b2bff81c3da0849db270"
8+
rev = "d7a1f9a81e19ed565d3c6a7c2dae586b6413c615"
99
package = "vrf-server"

0 commit comments

Comments
 (0)