feat(observability): direct-to-cloud OTLP with TLS + auth headers

AGENTS.md

@@ -58,7 +58,7 @@ The invariant index — what must stay true. Full narrative and rationale live i
 12. **Structured queries: column authz fail-closed (security)** — `POST /v1/query?table={table}`: typed AST validated against schema, permission-enforced, timestamp-bucketed for cache, `DefaultMaxRows` (10,000) cap. Every column reference — projection, aggregation args, `filters`, `group_by`, `order_by`, `time_range` — is authorized inside `query.Build` (the single chokepoint that enumerates them all), so no clause can skip the role's `allow_columns`/`deny_columns` check (#223). A `select_all` read by a *column-restricted* role expands to its allowed columns via `policy.AllowedProjection`, never a bare `SELECT *`; *unrestricted*/admin roles keep `SELECT *` (`policy.RestrictsColumns` decides). Omitting `columns` selects nothing (`ErrEmptyProjection` → `200 []`); `["*"]` is the literal column `*` (schema-gated, not a wildcard); a table-granted role with no readable columns fails closed (`ErrNoReadableColumns` → `403`). Structured and live-stream (`filterEventColumns`) reads share the one per-column decision `policy.IsColumnAllowed`, so column visibility can't drift. Preserve when touching `internal/query` or the structured-query handler. Detail: architecture.md § `query/`.
 13. **Named query pipes: fail-closed (security)** — pre-defined SQL templates (Tinybird-style) with param binding + caching; `GET/POST /v1/pipes/{name}` sit outside `RequireAdmin`, so per-pipe `allowed_roles` is the *only* execute-path gate, via `policy.RoleAllowed`: exact allowlist membership (no `"*"`), admin always passes, empty/absent role and empty-string entries authorize nobody, and no `allowed_roles` → admin-only. Preserve and exercise via `testutil.RunRoleMatrix` / `StandardRoleMatrix` (see #159). Detail: architecture.md § `pipes/`.
 14. **TypeScript SDK** — `@wavehouse/sdk`: zero-dep client, typed query builder, real-time SSE, live queries (incrementable/decomposable/poll aggregation), codegen CLI. The canonical client (see §SDK Sync).
-15. **Observability invariants** — stdout always 100% (sampling is OTLP-push-only); WARN+ERROR always export at 100% (a non-configurable floor — don't expose it); gRPC OTel exporters dial lazily so an unreachable collector never blocks startup; the OTel Prometheus exporter uses a **private** `prometheus.Registry`. Preserve when touching the logger/sampler/provider. Detail: architecture.md § `observability/`.
+15. **Observability invariants** — stdout always 100% (sampling is OTLP-push-only); WARN+ERROR always export at 100% (a non-configurable floor — don't expose it); gRPC OTel exporters dial lazily so an unreachable collector never blocks startup; the OTel Prometheus exporter uses a **private** `prometheus.Registry`. The OTLP endpoint/TLS/custom-CA/mTLS/headers are delegated to the OpenTelemetry SDK's standard `OTEL_EXPORTER_OTLP_*` env vars — `InitProvider` passes **no** endpoint/header options, with one exception: the gRPC logs exporter (`otlploggrpc` v0.19) loads `OTEL_EXPORTER_OTLP_CERTIFICATE`/client-cert vars but never wires them into its dial, so `logExporterTLSOptions` rebuilds the custom-CA/mTLS creds from those same env vars and passes them via `WithTLSCredentials` (traces/metrics on v1.43 need no help; drop the shim once the exporter honors the env TLS config upstream — a version bump alone isn't enough, v0.20 still ignores it). A malformed `OTEL_EXPORTER_OTLP_HEADERS` is logged and skipped by the SDK (fail-soft), not fatal. Preserve when touching the logger/sampler/provider. Detail: architecture.md § `observability/`.
 16. **Bearer-token-only CORS posture (security)** — Bearer JWT on every request, no cookies/sessions; `corsMiddleware` deliberately **never** emits `Access-Control-Allow-Credentials` (not needed, and `*` + credentials is a spec violation browsers reject). `cors_allowed_origins` controls who can *read* responses, not cookie scope; CSRF protection is structural. Don't reintroduce cookie auth or `Allow-Credentials` without a design discussion — answers GitHub #29/#30. Code: `internal/api/router.go`.
 17. **Non-fatal boot** — schema-discovery failure on boot is non-fatal: `cmd/wavehouse` records an `api.BootState`, binds `:8080`, serves 503 on `/livez`/`/readyz` with the diagnostic, and retries via `SchemaRegistry.RetryRefresh` (backoff 2s → 60s). Bounds supervisor restart loops.
 18. **Health endpoints** — liveness `/livez`, readiness `/readyz` (k8s convention); `/healthz` is a permanent alias of `/livez`; `/health` + `/ready` are deprecated (removal v0.2.0, CHANGELOG #144). `/v1/health` is the SDK's content-free public ping (no ClickHouse check), a `/v1` route so it survives reverse-proxy probe-path filtering. Point k8s at `/livez`/`/readyz`, SDK/online-checks at `/v1/health`, never the deprecated aliases.

CHANGELOG.md

@@ -67,6 +67,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
   - **Deferred to #196:** bounded-concurrency chunking of very large arrays + backpressure-aware retry. `insert([...])` currently sends one request regardless of size.
 - **npm publishing for `@wavehouse/sdk` via OIDC trusted publishing** (`.github/workflows/publish-npm.yml` (new), `.github/workflows/publish-dev.yml`, `clients/ts/package.json`, `clients/ts/LICENSE` (new)): closes #227. A dedicated `publish-npm.yml` is the single npm trusted publisher for the SDK (npm allows one trusted workflow file per package), with two jobs mirroring the Go/Docker release model: `dev` publishes `0.0.0-dev.<sha>` under the `dev` dist-tag on every push to `main` that touches `clients/ts/` (the npm analog of the `:dev` image), and `release` publishes a tagged version under `latest`/`alpha`/`beta`/`rc`/`next` (derived from the version) plus a GitHub Release on `sdk-v*` tags — decoupled from the server's `v*` tags. Auth is GitHub OIDC trusted publishing (no `NPM_TOKEN` secret); `--provenance` is deferred until the repo is public (provenance requires a public source repo). `clients/ts/package.json` gains `author`/`homepage`/`bugs` and a normalized `repository.url`, and the package ships a bundled `LICENSE`. `publish-dev.yml` is unchanged apart from a pointer comment.
 - **CORS support for the SSE streaming endpoint** (`internal/api/router.go`, `internal/api/stream.go`, `internal/api/router_test.go`, `docs/src/content/docs/api.md`): closes #215. `GET /v1/stream` already inherited the router-level `corsMiddleware` (`Access-Control-Allow-Origin` echo + OPTIONS preflight), but the SSE resumption header `Last-Event-ID` — which the handler reads to gap-fill from the client's last seen event — was not in `Access-Control-Allow-Headers`, so a cross-origin fetch-based SSE client resuming via that header failed its preflight. `Last-Event-ID` is now allow-listed, the SSE handler carries a comment pointing at the middleware, and new end-to-end tests pin CORS on `/v1/stream` (preflight advertises `Last-Event-ID`, the streaming GET echoes an allowed origin, a disallowed origin gets no header) so the behavior can't silently regress. Browser `EventSource` clients still pass the JWT via `?token=` since they can't set an `Authorization` header.
+- **Direct-to-cloud OTLP — TLS + auth headers via the standard `OTEL_EXPORTER_OTLP_*` env vars, no sidecar required** (`internal/observability/provider.go`, `internal/observability/logtls.go`, `internal/config/config.go`, `cmd/wavehouse/main.go`, `internal/testutil/otlp.go`, `tests/integration/otel_test.go`, `config.yaml`, `docs/src/content/docs/{configuration,deployment}.md`, `AGENTS.md`): closes [#97](https://github.com/Wave-RF/WaveHouse/issues/97). WaveHouse exports traces/metrics/logs straight to a cloud OTLP gateway (Honeycomb, Grafana Cloud) with no sidecar to terminate TLS or inject auth. The OTLP destination — endpoint, TLS (an `https://` endpoint selects system root CAs), a custom/private CA (`OTEL_EXPORTER_OTLP_CERTIFICATE`), mutual TLS (`OTEL_EXPORTER_OTLP_CLIENT_CERTIFICATE`/`_CLIENT_KEY`), and per-RPC auth headers (`OTEL_EXPORTER_OTLP_HEADERS`) — is configured through the OpenTelemetry SDK's standard `OTEL_EXPORTER_OTLP_*` environment variables; `InitProvider` passes no endpoint or header options, so custom-CA + mutual TLS work across all three signals. The one piece of WaveHouse glue is `logExporterTLSOptions`: the pinned gRPC logs exporter (`otlploggrpc` v0.19) loads `OTEL_EXPORTER_OTLP_CERTIFICATE` / client-cert vars but never wires them into its dial (only the programmatic `WithTLSCredentials` is honored), so without it the logs signal would silently fall back to system roots against a private-CA gateway; we rebuild those credentials from the same env vars and hand them to the log exporter (traces/metrics, on v1.43, need no help). A malformed `OTEL_EXPORTER_OTLP_HEADERS` entry is logged and skipped by the SDK (fail-soft), the same way the SDK handles every other `OTEL_EXPORTER_OTLP_*` value. The earlier `otel.addr`/`WH_OTEL_ADDR` and `otel.headers`/`WH_OTEL_HEADERS` config keys are removed in favor of the spec vars — set `OTEL_EXPORTER_OTLP_ENDPOINT` instead (default is the SDK's `localhost:4317`). Datadog has no public direct-to-cloud OTLP endpoint; its supported path remains the local DDOT Collector embedded in the Datadog Agent, reached as a plaintext receiver (see the deployment guide). Integration tests pin the TLS dial path — trusting an in-process self-signed receiver via `OTEL_EXPORTER_OTLP_CERTIFICATE`, the same path a real operator uses — and header propagation across all signals.
 - **`GET /version` build-info endpoint** (`internal/api/version.go` (new), `internal/api/version_test.go` (new), `internal/api/router.go`, `internal/api/router_test.go`, `cmd/wavehouse/main.go`, `docs/src/content/docs/api.md`): a public, unauthenticated endpoint returning the binary's `version`, `git_commit`, and `build_time` — the same values injected via `-ldflags` (`Makefile`, `.goreleaser.yaml`) and logged at startup — plus `go_version` from `runtime.Version()`. Completes issue #44: the ldflags injection and startup logging already shipped; this exposes the same metadata over HTTP so an operator can confirm exactly which build is running on a deployed instance. Wired alongside `/livez` and `/readyz`, outside the `/v1` auth tree; the values are non-sensitive and already in the logs, so there is no auth gate.
 - **Configurable RBAC: a single admin role, public access driven by the policy, and authentication decoupled from authorization** (`internal/auth/`, `internal/policy/`, `internal/api/`, `internal/config/config.go`, `cmd/wavehouse/main.go`, `config.yaml`, `docs/src/content/docs/configuration.md`, `docs/src/content/docs/api.md`, `AGENTS.md`):
   - **No `auth.enabled` / `auth.dev_mode` flags.** The JWT middleware (new `internal/auth` package) always runs. A request with **no** token, or an **invalid/expired/malformed** one, falls back to an empty role that `policy.ResolveRole` maps to the policy `default_role`; the token error is recorded in the request context so a gate that ultimately denies **fails loud** (`401` "invalid/expired token") rather than a bare `403`. Elevated access requires a valid token. With neither `jwt_secret` nor `jwks_url` configured, no token validates and every request is the `default_role` — a pure public deployment.

cmd/wavehouse/main.go

@@ -104,8 +104,11 @@ func run() int {
 	// wanted — Prometheus-only operation (Alloy/scrape, no collector) is a
 	// first-class mode. The OTel SDK MeterProvider is the shared substrate.
 	if cfg.OTel.Enabled || cfg.Prometheus.Enabled {
+		// Endpoint, TLS, and auth headers come from the standard
+		// OTEL_EXPORTER_OTLP_* env vars, read by the SDK. A malformed header is
+		// logged and skipped by the SDK (fail-soft); InitProvider's own error is
+		// likewise non-fatal — we log it and fall back to stdout below.
 		otelShutdown, ph, err := observability.InitProvider(ctx, serviceName, observability.ProviderConfig{
-			Endpoint:          cfg.OTel.Addr,
 			TracesEnabled:     cfg.OTel.Enabled && cfg.OTel.Traces.Enabled,
 			TracesSampleRate:  cfg.OTel.Traces.SampleRate,
 			MetricsEnabled:    cfg.OTel.Enabled && cfg.OTel.Metrics.Enabled,
@@ -136,11 +139,15 @@ func run() int {
 				)
 				slog.SetDefault(logger)
 			}
+			otlpEndpoint := os.Getenv("OTEL_EXPORTER_OTLP_ENDPOINT")
+			if otlpEndpoint == "" {
+				otlpEndpoint = "localhost:4317 (SDK default)"
+			}
 			switch {
 			case cfg.OTel.Enabled && cfg.Prometheus.Enabled:
-				logger.Info("observability pipeline established", "otlp_endpoint", cfg.OTel.Addr, "prometheus", true)
+				logger.Info("observability pipeline established", "otlp_endpoint", otlpEndpoint, "prometheus", true)
 			case cfg.OTel.Enabled:
-				logger.Info("observability pipeline established", "otlp_endpoint", cfg.OTel.Addr)
+				logger.Info("observability pipeline established", "otlp_endpoint", otlpEndpoint)
 			case cfg.Prometheus.Enabled:
 				logger.Info("observability pipeline established", "prometheus", true)
 			}

config.yaml

@@ -19,7 +19,9 @@ server:
 
 otel:
   enabled: false         # master switch — set true to export via OTLP gRPC
-  addr: "127.0.0.1:4317"
+  # Endpoint, TLS, custom CA, mTLS, and auth headers come from the standard
+  # OTEL_EXPORTER_OTLP_* env vars (OTEL_EXPORTER_OTLP_ENDPOINT=https://host:port,
+  # OTEL_EXPORTER_OTLP_HEADERS=x-honeycomb-team=KEY), read by the OTel SDK.
   traces:
     enabled: true
     sample_rate: 1.0     # head-based, [0.0, 1.0]; tune down for high QPS

docs/src/content/docs/configuration.md

@@ -115,12 +115,11 @@ The master switch is `otel.enabled`. When `true`, each signal (traces/metrics/lo
 
 **Sampling rates apply only to the OTLP push path.** Stdout always emits 100% of records — operators using a scraping-style pipeline (Promtail/Grafana Alloy → Loki, Vector, Fluent Bit, etc.) set the collection rate at the scraper, not the application. WaveHouse pushes telemetry to an OTel collector; the scraper world owns its own ingest policy. If you want to throttle OTLP volume for cost, lower the rates below. If you want to throttle Loki/Datadog Logs/etc., do it at that pipeline.
 
-**TLS / direct-to-cloud limitation.** The OTLP exporters currently use `WithInsecure()` — plaintext gRPC only. WaveHouse cannot ship directly to TLS-protected OTLP endpoints (Grafana Cloud's OTLP gateway, Honeycomb, Datadog OTLP, etc.). The standard workaround is a sidecar collector (the OTel collector or Grafana Alloy) on `127.0.0.1:4317` that receives our plaintext OTLP and re-exports to the cloud endpoint with TLS + auth headers configured locally. Tracked in #97.
+**Direct-to-cloud OTLP.** The OTLP destination is configured through the standard [`OTEL_EXPORTER_OTLP_*` environment variables](https://opentelemetry.io/docs/specs/otel/protocol/exporter/) read by the OpenTelemetry SDK, not WaveHouse config: `OTEL_EXPORTER_OTLP_ENDPOINT` (an `https://` URL selects TLS via system root CAs; `http://` or a bare host stays plaintext — default `localhost:4317`), `OTEL_EXPORTER_OTLP_HEADERS` for per-RPC auth, `OTEL_EXPORTER_OTLP_CERTIFICATE` to trust a custom/private CA, and `OTEL_EXPORTER_OTLP_CLIENT_CERTIFICATE` / `OTEL_EXPORTER_OTLP_CLIENT_KEY` for mutual TLS. With these set, WaveHouse ships telemetry straight to a TLS-protected cloud gateway (Grafana Cloud's OTLP gateway, Honeycomb, etc.) — no sidecar required (a sidecar is still useful for egress queuing, batching, and tail-based sampling; it's just no longer mandatory). A malformed `OTEL_EXPORTER_OTLP_HEADERS` entry is logged and skipped by the OpenTelemetry SDK (fail-soft) rather than failing startup. Datadog has no public direct-to-cloud OTLP endpoint — use the local DDOT Collector path in the [deployment guide](/deployment#observability). See [Deployment → Observability](/deployment#observability) for worked Honeycomb / Grafana Cloud examples.
 
 | YAML Key | Env Var | Default | Description |
 | -------- | ------- | ------- | ----------- |
-| `otel.enabled` | `WH_OTEL_ENABLED` | `false` | Master switch. When `false`, no signals are initialized regardless of the sub-toggles below. |
-| `otel.addr` | `WH_OTEL_ADDR` | `127.0.0.1:4317` | OTLP gRPC endpoint used by every enabled signal. Plain `host:port` — no scheme, plaintext gRPC only (see TLS note above). See `make help` for a local collector setup. |
+| `otel.enabled` | `WH_OTEL_ENABLED` | `false` | Master switch. When `false`, no signals are initialized regardless of the sub-toggles below. The OTLP endpoint, TLS, custom CA, mutual TLS, and auth headers are configured via the standard `OTEL_EXPORTER_OTLP_*` env vars (see the note above), not a WaveHouse key. |
 | `otel.traces.enabled` | `WH_OTEL_TRACES_ENABLED` | `true` | Export traces via OTLP gRPC. |
 | `otel.traces.sample_rate` | `WH_OTEL_TRACES_SAMPLE_RATE` | `1.0` | Head-based trace sampling rate in `[0.0, 1.0]`. `1.0` exports every trace; `0.0` exports none. Defaults to 100% (matches the OpenTelemetry SDK default); lower it for high-QPS production services where collector or backend cost is a concern. Best practice is "100% at the source, downsample at the collector" via tail-based sampling. Validated at config load. |
 | `otel.metrics.enabled` | `WH_OTEL_METRICS_ENABLED` | `true` | Export metrics + Go runtime metrics via OTLP gRPC. Periodic reader interval is fixed at 15s. Metrics are pre-aggregated so there is no sampling knob. |