From 23eb06602a0c924ccbbcaadcacdfd3cf9fe9cc38 Mon Sep 17 00:00:00 2001 From: Jack Woods Date: Wed, 13 May 2026 12:27:46 -0400 Subject: [PATCH 01/25] =?UTF-8?q?feat(observability):=20direct-to-cloud=20?= =?UTF-8?q?OTLP=20=E2=80=94=20TLS=20+=20auth=20headers=20(#97)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Closes #97. The OTLP exporters previously hardcoded WithInsecure(), so WaveHouse could only push to a plaintext-gRPC receiver — practically, a local sidecar collector or Alloy. Operators wanting to ship straight to Grafana Cloud, Honeycomb, or Datadog had to run a sidecar to terminate TLS + add the per-RPC auth header. This commit closes that gap. Config — four new fields, all defaults preserve current behavior: - otel.headers (WH_OTEL_HEADERS) — OTel-spec env-var format, comma-separated `key=value`. Values may contain `=` so base64 padding round-trips. Applied as gRPC metadata to every exporter. - otel.{traces,metrics,logs}.addr — optional per-signal endpoint override (Grafana Cloud uses distinct gateway hosts per signal). Empty means inherit otel.addr. Provider — scheme-aware endpoint: - `https://` → TLS via system root CAs (credentials.NewTLS). - `http://` or bare `host:port` → plaintext (backward-compat). - URL path component stripped (gRPC ignores it). - Headers applied uniformly to all three exporters (no per-signal header override — per-signal endpoint is the override knob). - ProviderConfig.TLSConfig is a test-only escape hatch; production leaves it nil for system-roots TLS. The FakeOTLPTLS integration test injects an ephemeral-cert tls.Config through this field. Validation — Validate() rejects malformed headers at boot (missing `=`, empty key) so a typo can't silently disable auth in production. The check is skipped when otel.enabled=false to keep yaml-iteration ergonomic. Tests: - Unit (internal/observability/endpoint_test.go): table-driven coverage of ParseEndpoint (all scheme cases incl. path stripping) and ParseOTelHeaders (whitespace, base64-with-trailing-=, malformed rejection). - Config: TestValidate_RejectsMalformedHeaders, TestValidate_AcceptsValidHeaders, TestValidate_HeadersIgnoredWhenOTelDisabled, plus defaults coverage. - Integration: TestOTel_TLSPath_Traces (FakeOTLPTLS with ephemeral ECDSA self-signed cert), TestOTel_Headers_AppliedToAllSignals (asserts authorization header on traces/metrics/logs via captured gRPC metadata), TestOTel_PerSignalEndpoint_Override (split traces and metrics across two receivers). FakeOTLP — NewFakeOTLPTLS(t) constructor + TLSConfig() accessor for the matching client config; gRPC metadata captured per-signal via new LastTraceHeaders / LastMetricHeaders / LastLogHeaders accessors. Existing plaintext NewFakeOTLP path unchanged. Docs — configuration.md gets the four new env-var rows and a rewritten TLS callout (the prior "limitation" note is gone). deployment.md adds a Direct-to-cloud subsection with worked examples for Honeycomb, Grafana Cloud (Basic auth via base64), and Datadog OTLP. AGENTS.md key design decision #15 grows one sentence noting that scheme-sniffing is the supported TLS knob (no separate `tls.enabled` boolean) and that per-signal endpoint is the override knob (not per-signal headers). Out of scope: mTLS / client-certificate auth. Co-Authored-By: Claude Opus 4.7 (1M context) --- AGENTS.md | 2 +- CHANGELOG.md | 1 + cmd/wavehouse/main.go | 7 ++ docs/src/content/docs/configuration.md | 10 +- docs/src/content/docs/deployment.md | 43 ++++++- internal/config/config.go | 50 ++++++++- internal/config/config_test.go | 63 +++++++++++ internal/observability/endpoint.go | 76 +++++++++++++ internal/observability/endpoint_test.go | 65 +++++++++++ internal/observability/provider.go | 75 ++++++++++--- internal/testutil/otlp.go | 142 ++++++++++++++++++++++-- tests/integration/otel_test.go | 118 ++++++++++++++++++++ 12 files changed, 623 insertions(+), 29 deletions(-) create mode 100644 internal/observability/endpoint.go create mode 100644 internal/observability/endpoint_test.go diff --git a/AGENTS.md b/AGENTS.md index 3232dfb2..fbedb7fd 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -42,7 +42,7 @@ Twelve internal packages under `internal/`: 12. **Structured queries**: Type-safe query AST endpoint (`POST /v1/tables/{table}/query`) validated against schema, with permission enforcement, timestamp bucketing for cache optimization, and `DefaultMaxRows` (10,000) limit cap. 13. **Named query pipes**: Pre-defined SQL templates (inspired by Tinybird) with parameter binding, role restrictions, and caching. Stored in NATS KV with `.sql` file directory bootstrap. 14. **TypeScript SDK**: `@wavehouse/sdk` — zero-dependency client with typed query builder, real-time SSE, live queries with smart aggregation classification (incrementable/decomposable/poll), and codegen CLI. -15. **Observability invariants**: Stdout is *always* 100% — the slog logger fans out to stdout AND OTLP, and stdout sampling would silently hide records that scraping pipelines (Promtail/Alloy/Vector → Loki) are paying to store. Sampling knobs apply only to OTLP push. WARN+ERROR records always export at 100% regardless of `logs.sample_rate` — silently dropping errors during incidents would be a worse failure mode than the cost of forwarding them all (this is a *non-configurable* floor; do not expose it). gRPC OTel exporters dial lazily, so an unreachable collector never blocks startup; transient failures surface via the OTel SDK's error handler. The OTel Prometheus exporter (when enabled) uses a *private* `prometheus.Registry` to avoid leaking process/Go collectors that `prometheus.DefaultRegisterer` auto-registers into our `/metrics` output. When changing the logger, the sampler, or the provider wiring, preserve these invariants. +15. **Observability invariants**: Stdout is *always* 100% — the slog logger fans out to stdout AND OTLP, and stdout sampling would silently hide records that scraping pipelines (Promtail/Alloy/Vector → Loki) are paying to store. Sampling knobs apply only to OTLP push. WARN+ERROR records always export at 100% regardless of `logs.sample_rate` — silently dropping errors during incidents would be a worse failure mode than the cost of forwarding them all (this is a *non-configurable* floor; do not expose it). gRPC OTel exporters dial lazily, so an unreachable collector never blocks startup; transient failures surface via the OTel SDK's error handler. The OTel Prometheus exporter (when enabled) uses a *private* `prometheus.Registry` to avoid leaking process/Go collectors that `prometheus.DefaultRegisterer` auto-registers into our `/metrics` output. TLS is selected via *scheme sniffing* on `otel.addr` (`https://` → TLS, anything else → plaintext) — there is no separate `otel.tls.enabled` boolean. `otel.headers` is applied uniformly to all OTLP exporters (no per-signal header override); per-signal endpoint overrides (`otel.{traces,metrics,logs}.addr`) are the supported way to split signals across gateways. When changing the logger, the sampler, or the provider wiring, preserve these invariants. 16. **Bearer-token-only CORS posture**: WaveHouse is a Bearer-token API — `Authorization: Bearer ` on every authenticated request, no cookies, no session middleware. The CORS middleware (`internal/api/router.go` `corsMiddleware`) deliberately **never** emits `Access-Control-Allow-Credentials`, because (a) we don't need it (Bearer tokens are explicit request headers, not browser-managed credentials) and (b) the historical pairing of `Allow-Credentials: true` with `Allow-Origin: *` is a CORS spec violation that browsers reject. The `cors_allowed_origins` allowlist controls *which origins can read responses*, not cookie scope. CSRF protection is structural: cross-site requests can't smuggle a Bearer token because the browser won't auto-attach `Authorization` headers cross-origin. Do not reintroduce cookie-based auth or `Access-Control-Allow-Credentials` without a separate design discussion — the current posture is the answer to GitHub issues #29 and #30. ## Code Conventions diff --git a/CHANGELOG.md b/CHANGELOG.md index 19e05ae5..ae675cdc 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - **Dependabot auto-merge no longer slipped past CI on consolidated-pipeline PRs** (`main branch protection` ruleset; `.github/workflows/project-orchestrator.yml`; `.github/workflows/claude-review.yml`; `AGENTS.md`): on commit `93b3206` the previously six-job CI was collapsed into a single job named `CI`, but three places still referenced the old multi-job names — the ruleset's `required_status_checks` (`Validate` + `Admin approval` only — `CI` not listed), the `bot-clean` check in `project-orchestrator.yml` (looking for `Build`/`Validate`/`Lint`/`Test`/`Integration Tests`/`SDK Tests`), and `REQUIRED_CHECKS` in `claude-review.yml` (same six). Net effect: a Dependabot PR opened at T+0 would have its `Validate` and `Admin approval` checks satisfied within ~30s (PR-title workflow + the auto-approval workflow), `gh pr merge --auto` would fire, GitHub would see all *required* checks green, and the PR would squash-merge before `ci.yml` even started running. Pre-consolidation runs (PR #90 etc.) didn't expose this because GitHub's auto-merge waits for in-flight checks as a courtesy, and the multi-job CI was producing checks before auto-merge fired — but with one collapsed `CI` check that didn't start until later, the courtesy wait disappeared. Fix: added `CI` to the ruleset's required-checks (now `Admin approval` / `CI` / `Validate`) via the `gh api PUT /repos/Wave-RF/WaveHouse/rulesets/15353356` round-trip, and updated both bot-clean lists to look for `CI` + `Validate`. AGENTS.md note about the required-check set updated to match. Diagnosis steps recorded inline as code comments in both workflow files so future-me knows where to look if the check name changes again. ### Added +- **Direct-to-cloud OTLP — TLS + auth headers, no sidecar required** (`internal/config/config.go`, `internal/observability/provider.go`, `internal/observability/endpoint.go`, `internal/testutil/otlp.go`, `cmd/wavehouse/main.go`, `tests/integration/otel_test.go`, `docs/src/content/docs/{configuration,deployment}.md`, `AGENTS.md`): closes [#97](https://github.com/Wave-RF/WaveHouse/issues/97). `otel.addr` is now scheme-aware — `https://` selects TLS (system root CAs by default), `http://` or a bare `host:port` stays plaintext gRPC (backward-compat with the prior `WithInsecure()` default). New `otel.headers` (`WH_OTEL_HEADERS`) takes the OTel spec env-var format — comma-separated `key=value`, values may contain `=` so base64 padding round-trips — and applies the parsed map as gRPC metadata to every OTLP exporter (traces, metrics, logs). Together these unlock direct push to Honeycomb / Grafana Cloud / Datadog OTLP without a sidecar. Per-signal endpoint overrides (`otel.{traces,metrics,logs}.addr`) point one signal at a different gateway than the top-level default (Grafana Cloud uses distinct trace/metric/log hosts in some regions); empty means inherit. Headers always apply uniformly — there is intentionally no per-signal header override. Test surface: `FakeOTLP` gains a `NewFakeOTLPTLS(t)` constructor (ephemeral self-signed cert + matching client `*tls.Config`) and per-signal `LastTraceHeaders / LastMetricHeaders / LastLogHeaders` accessors via captured gRPC metadata. New integration tests pin the TLS dial path, headers-on-all-signals propagation, and per-signal endpoint splitting. Validation rejects malformed `otel.headers` (missing `=`, empty key) at boot rather than letting a typo silently disable auth in production. mTLS / client-certificate auth is not yet supported. - **Astro / Starlight documentation site at `wavehouse.dev`** (`docs/`, `Makefile`, `.github/dependabot.yml`, `.github/workflows/ci.yml`, `.gitignore`, `.vscode/launch.json`): full content + tooling for the public docs (Getting Started, Why WaveHouse, Architecture, API Reference, TypeScript SDK, Configuration, Deployment, Development). Build-time mermaid via `rehype-mermaid` (inline-svg strategy, no client-side JS), LaTeX via `remark-math` + `rehype-katex`, image zoom via `starlight-image-zoom`, expressiveCode with github dark/light + `env`/`dns` Shiki language aliases. PostHog frontend telemetry (key is public by design — embedded in every visitor's browser). `editLink` pointed at `main`; `lastUpdated` on. Sidebar lives in `docs/src/config/sidebar.ts` as the single source of truth, consumed by both Starlight rendering and the LLM-friendly outputs below. Cloudflare Workers + Static Assets deployment via `docs/wrangler.jsonc` (no auto-deploy yet — wired up when the GH Actions workflow lands separately). Make targets stay minimal: `dev-docs` (Astro dev server on :4321), `build-docs`, `preview-docs` — the last serves the production build through `wrangler dev`, so the Cloudflare Worker's content negotiation (`Accept: text/markdown` / LLM-bot User-Agent → `.md` twin) is exercised end-to-end the same way it will be in production. - **Per-page `.md` twin + `llms.txt` family via `starlight-llm-tools`** (`docs/astro.config.mjs`, `docs/worker/index.ts`, `docs/wrangler.jsonc`): every doc is also served as raw markdown at `.md` with a navigation header (Section / Subpages / Related / HTML version pointers), plus three concatenated views — `/llms.txt` manifest, `/llms-full.txt` (every page, sidebar-ordered), `/llms-small.txt` (overview pages only). Page-title slot gains a **Copy Markdown** button and **Open with AI** dropdown (Claude / ChatGPT / Cursor). Content negotiation is handled by a Cloudflare Worker that re-exports the `cloudflare-md-router` package's default handler in one line — `Accept: text/markdown` or known LLM-bot User-Agent (GPTBot, ClaudeBot, PerplexityBot, Applebot-Extended, etc.) gets the `.md` twin transparently, with a fall-through to the HTML response when the twin doesn't exist. - **Two extracted plugin packages, MIT-licensed**: [`github.com/Wave-RF/cloudflare-md-router`](https://github.com/Wave-RF/cloudflare-md-router) (Workers handler + `createMdRouter()` factory + extensible bot-UA regex; consumed in `docs/worker/index.ts` as a one-line re-export) and [`github.com/Wave-RF/starlight-llm-tools`](https://github.com/Wave-RF/starlight-llm-tools) (Starlight plugin that auto-injects the four routes, the two components, and the `PageTitle` override; optionally calls into `starlight-glossary/transform` when present). Both pinned via `github:Wave-RF/...` in `docs/package.json`; lockfile pins the resolved commit SHAs. Pre-npm-publish state — switch to normal version specifiers when the packages land on npm. diff --git a/cmd/wavehouse/main.go b/cmd/wavehouse/main.go index cbdccc71..90fab7cf 100644 --- a/cmd/wavehouse/main.go +++ b/cmd/wavehouse/main.go @@ -102,12 +102,19 @@ func run() int { // wanted — Prometheus-only operation (Alloy/scrape, no collector) is a // first-class mode. The OTel SDK MeterProvider is the shared substrate. if cfg.OTel.Enabled || cfg.Prometheus.Enabled { + // Headers already passed config validation; the parser cannot fail + // here for input that satisfied Validate(). + headers, _ := observability.ParseOTelHeaders(cfg.OTel.Headers) otelShutdown, ph, err := observability.InitProvider(ctx, serviceName, observability.ProviderConfig{ Endpoint: cfg.OTel.Addr, + Headers: headers, + TracesEndpoint: cfg.OTel.Traces.Addr, TracesEnabled: cfg.OTel.Enabled && cfg.OTel.Traces.Enabled, TracesSampleRate: cfg.OTel.Traces.SampleRate, + MetricsEndpoint: cfg.OTel.Metrics.Addr, MetricsEnabled: cfg.OTel.Enabled && cfg.OTel.Metrics.Enabled, PrometheusEnabled: cfg.Prometheus.Enabled, + LogsEndpoint: cfg.OTel.Logs.Addr, LogsEnabled: cfg.OTel.Enabled && cfg.OTel.Logs.Enabled, }) if err != nil { diff --git a/docs/src/content/docs/configuration.md b/docs/src/content/docs/configuration.md index 2bfbd6cd..d963f095 100644 --- a/docs/src/content/docs/configuration.md +++ b/docs/src/content/docs/configuration.md @@ -113,16 +113,22 @@ The master switch is `otel.enabled`. When `true`, each signal (traces/metrics/lo **Sampling rates apply only to the OTLP push path.** Stdout always emits 100% of records — operators using a scraping-style pipeline (Promtail/Grafana Alloy → Loki, Vector, Fluent Bit, etc.) set the collection rate at the scraper, not the application. WaveHouse pushes telemetry to an OTel collector; the scraper world owns its own ingest policy. If you want to throttle OTLP volume for cost, lower the rates below. If you want to throttle Loki/Datadog Logs/etc., do it at that pipeline. -**TLS / direct-to-cloud limitation.** The OTLP exporters currently use `WithInsecure()` — plaintext gRPC only. WaveHouse cannot ship directly to TLS-protected OTLP endpoints (Grafana Cloud's OTLP gateway, Honeycomb, Datadog OTLP, etc.). The standard workaround is a sidecar collector (the OTel collector or Grafana Alloy) on `127.0.0.1:4317` that receives our plaintext OTLP and re-exports to the cloud endpoint with TLS + auth headers configured locally. Tracked in #97. +**TLS + direct-to-cloud OTLP.** `otel.addr` is scheme-aware: `https://` selects TLS (system root CAs), `http://` or a bare `host:port` stays plaintext gRPC. Combined with `otel.headers` for auth, WaveHouse can ship telemetry straight to TLS-protected cloud endpoints (Grafana Cloud's OTLP gateway, Honeycomb, Datadog OTLP, etc.) without a sidecar collector. A sidecar is still useful for egress queuing, batching, and tail-based sampling — it's just no longer required. See [Deployment → Direct-to-cloud OTLP](./deployment.md#direct-to-cloud-otlp) for worked examples. + +If different signals need different gateway hosts (Grafana Cloud's distinct trace / metric / log endpoints, say), set `otel.{traces,metrics,logs}.addr` to override the default for that signal — empty means inherit from `otel.addr`. | YAML Key | Env Var | Default | Description | | -------- | ------- | ------- | ----------- | | `otel.enabled` | `WH_OTEL_ENABLED` | `false` | Master switch. When `false`, no signals are initialized regardless of the sub-toggles below. | -| `otel.addr` | `WH_OTEL_ADDR` | `127.0.0.1:4317` | OTLP gRPC endpoint used by every enabled signal. Plain `host:port` — no scheme, plaintext gRPC only (see TLS note above). See `deployments/signoz/` for a local collector setup. | +| `otel.addr` | `WH_OTEL_ADDR` | `127.0.0.1:4317` | Default OTLP gRPC endpoint. Accepts `host:port` (plaintext), `http://host:port` (plaintext), or `https://host:port` (TLS via system root CAs). A trailing URL path is tolerated and stripped — gRPC routes by service name. See `deployments/signoz/` for a local plaintext collector setup. | +| `otel.headers` | `WH_OTEL_HEADERS` | *(empty)* | Comma-separated `key=value` pairs applied as gRPC metadata to every OTLP export — the standard auth knob for cloud endpoints (`authorization=Basic `, `x-honeycomb-team=`). Values may contain `=`; only the first `=` per segment splits key from value (base64 padding round-trips). Validated at config load. | | `otel.traces.enabled` | `WH_OTEL_TRACES_ENABLED` | `true` | Export traces via OTLP gRPC. | +| `otel.traces.addr` | `WH_OTEL_TRACES_ADDR` | *(empty)* | Per-signal override for `otel.addr` (same scheme rules). Empty means inherit. | | `otel.traces.sample_rate` | `WH_OTEL_TRACES_SAMPLE_RATE` | `1.0` | Head-based trace sampling rate in `[0.0, 1.0]`. `1.0` exports every trace; `0.0` exports none. Defaults to 100% (matches the OpenTelemetry SDK default); lower it for high-QPS production services where collector or backend cost is a concern. Best practice is "100% at the source, downsample at the collector" via tail-based sampling. Validated at config load. | | `otel.metrics.enabled` | `WH_OTEL_METRICS_ENABLED` | `true` | Export metrics + Go runtime metrics via OTLP gRPC. Periodic reader interval is fixed at 15s. Metrics are pre-aggregated so there is no sampling knob. | +| `otel.metrics.addr` | `WH_OTEL_METRICS_ADDR` | *(empty)* | Per-signal override for `otel.addr`. Empty means inherit. | | `otel.logs.enabled` | `WH_OTEL_LOGS_ENABLED` | `true` | Export logs via OTLP gRPC. Disabling this leaves stdout logging untouched — the OTel logger provider is simply not registered. | +| `otel.logs.addr` | `WH_OTEL_LOGS_ADDR` | *(empty)* | Per-signal override for `otel.addr`. Empty means inherit. | | `otel.logs.sample_rate` | `WH_OTEL_LOGS_SAMPLE_RATE` | `1.0` | OTLP export rate for `DEBUG`/`INFO` records, in `[0.0, 1.0]`. Validated at config load. `WARN` and `ERROR` records always export at 100% — dropping them silently during incidents is too dangerous to expose as a knob. **Stdout always receives 100% of records regardless of this rate** (see the scraper note above). | ### Prometheus diff --git a/docs/src/content/docs/deployment.md b/docs/src/content/docs/deployment.md index 2f6b8bd4..6f5a2ac8 100644 --- a/docs/src/content/docs/deployment.md +++ b/docs/src/content/docs/deployment.md @@ -314,9 +314,48 @@ Set `otel.enabled: true` (or `WH_OTEL_ENABLED=true`) and point `otel.addr` at th WaveHouse **pushes** to an OTel collector; scraping-style pipelines (Promtail/Grafana Alloy → Loki, Vector, Fluent Bit) read stdout directly and own their own sample rates. The `otel.{traces,logs}.sample_rate` knobs apply only to the OTLP push path. Stdout always emits 100%. The logger fans out to both stdout and OTLP, so stdout output never disappears regardless of collector state. gRPC exporters are lazy, so an unreachable collector does not block startup — transient export errors are surfaced via the OTel SDK's error handler instead. -### Pattern: SigNoz / Honeycomb / OTel-native backends +### Pattern: SigNoz / OTel-native backends (local collector) -Point `otel.addr` at the OTLP gRPC endpoint. All three signals (traces, metrics, logs) push through the same connection. This is the default and the simplest setup. +Point `otel.addr` at a plaintext OTLP gRPC endpoint. All three signals (traces, metrics, logs) push through the same connection. This is the default and the simplest setup. + +```yaml +otel: + enabled: true + addr: 127.0.0.1:4317 # bare host:port — plaintext gRPC +``` + +### Pattern: Direct-to-cloud OTLP (Honeycomb, Grafana Cloud, Datadog OTLP, etc.) + +`otel.addr` is scheme-aware: `https://` selects TLS, `http://` or no scheme stays plaintext. Combine with `otel.headers` for the per-RPC auth that every cloud OTLP gateway expects. No sidecar required — a sidecar is still useful for egress queuing, batching, and tail-based sampling, but is no longer the only way to get TLS + auth. + +**Honeycomb (single endpoint, per-RPC auth):** + +```bash +export WH_OTEL_ENABLED=true +export WH_OTEL_ADDR=https://api.honeycomb.io:443 +export WH_OTEL_HEADERS=x-honeycomb-team=YOUR_API_KEY +``` + +**Grafana Cloud OTLP gateway (Basic auth):** + +```bash +export WH_OTEL_ENABLED=true +export WH_OTEL_ADDR=https://otlp-gateway-prod-us-east-0.grafana.net:443 +# instanceID:token, base64-encoded; backslash-escape the space if your shell needs it +export WH_OTEL_HEADERS=authorization=Basic $(printf '%s' "$INSTANCE_ID:$TOKEN" | base64) +``` + +**Datadog OTLP intake:** + +```bash +export WH_OTEL_ENABLED=true +export WH_OTEL_ADDR=https://otlp.datadoghq.com:4317 +export WH_OTEL_HEADERS=dd-api-key=YOUR_API_KEY +``` + +If different signals need different gateway hosts (Grafana Cloud's traces/metrics/logs endpoints differ in some regions), set `WH_OTEL_TRACES_ADDR` / `WH_OTEL_METRICS_ADDR` / `WH_OTEL_LOGS_ADDR` to override the default per signal. Empty means inherit from `otel.addr`. Headers always apply to every exporter — there is intentionally no per-signal header override. + +mTLS / client-certificate auth is not yet supported; open an issue if you need it. ### Pattern: Grafana Cloud / Mimir / Loki / Tempo via Grafana Alloy diff --git a/internal/config/config.go b/internal/config/config.go index dee0695c..2a66afd4 100644 --- a/internal/config/config.go +++ b/internal/config/config.go @@ -32,9 +32,23 @@ type Config struct { // OTel configures the OpenTelemetry pipeline. `enabled` is the master switch; // when false, no signals are initialized regardless of the per-signal toggles. +// +// Addr accepts plain `host:port` (plaintext gRPC, backward-compat), `http://` +// (also plaintext), or `https://` (TLS — required for direct-to-cloud OTLP). +// A URL path component is tolerated and stripped (gRPC ignores it). +// +// Headers is a comma-separated list of `key=value` pairs applied to every OTLP +// exporter — the standard knob for auth against cloud endpoints +// (`authorization=Basic `, `x-honeycomb-team=`, etc.). Values may +// contain `=` (only the first `=` per segment splits key from value). +// +// Per-signal Addr overrides (Traces.Addr, Metrics.Addr, Logs.Addr) point one +// signal at a different endpoint than the top-level Addr — useful for Grafana +// Cloud, which uses distinct gateway hosts per signal. Empty means inherit. type OTel struct { Enabled bool `yaml:"enabled" env:"WH_OTEL_ENABLED" env-default:"false"` Addr string `yaml:"addr" env:"WH_OTEL_ADDR" env-default:"127.0.0.1:4317"` + Headers string `yaml:"headers" env:"WH_OTEL_HEADERS"` Traces OTelTraces `yaml:"traces"` Metrics OTelMetrics `yaml:"metrics"` Logs OTelLogs `yaml:"logs"` @@ -43,10 +57,12 @@ type OTel struct { type OTelTraces struct { Enabled bool `yaml:"enabled" env:"WH_OTEL_TRACES_ENABLED" env-default:"true"` SampleRate float64 `yaml:"sample_rate" env:"WH_OTEL_TRACES_SAMPLE_RATE" env-default:"1.0"` + Addr string `yaml:"addr" env:"WH_OTEL_TRACES_ADDR"` } type OTelMetrics struct { - Enabled bool `yaml:"enabled" env:"WH_OTEL_METRICS_ENABLED" env-default:"true"` + Enabled bool `yaml:"enabled" env:"WH_OTEL_METRICS_ENABLED" env-default:"true"` + Addr string `yaml:"addr" env:"WH_OTEL_METRICS_ADDR"` } // Prometheus controls a Prometheus exposition endpoint served alongside (or @@ -78,6 +94,7 @@ type Prometheus struct { type OTelLogs struct { Enabled bool `yaml:"enabled" env:"WH_OTEL_LOGS_ENABLED" env-default:"true"` SampleRate float64 `yaml:"sample_rate" env:"WH_OTEL_LOGS_SAMPLE_RATE" env-default:"1.0"` + Addr string `yaml:"addr" env:"WH_OTEL_LOGS_ADDR"` } type Server struct { @@ -146,6 +163,30 @@ type DLQ struct { Enabled bool `yaml:"enabled" env:"WH_DLQ_ENABLED" env-default:"true"` } +// validateOTelHeaders mirrors observability.ParseOTelHeaders for boot-time +// validation. Kept here (rather than importing observability) so config stays +// at the bottom of the dependency graph. +func validateOTelHeaders(s string) error { + s = strings.TrimSpace(s) + if s == "" { + return nil + } + for _, seg := range strings.Split(s, ",") { + seg = strings.TrimSpace(seg) + if seg == "" { + continue + } + i := strings.IndexByte(seg, '=') + if i < 0 { + return fmt.Errorf("header segment %q missing '='", seg) + } + if strings.TrimSpace(seg[:i]) == "" { + return fmt.Errorf("header segment %q has empty key", seg) + } + } + return nil +} + // Validate checks the loaded configuration for logical consistency. func (c *Config) Validate() error { if c.ClickHouse.HTTPScheme != "http" && c.ClickHouse.HTTPScheme != "https" { @@ -192,6 +233,13 @@ func (c *Config) Validate() error { return fmt.Errorf("otel.logs.sample_rate %g out of range [0.0, 1.0]", r) } } + // Parse headers eagerly so a malformed entry fails at boot rather than + // silently dropping auth in production. The same parser runs again at + // provider init; duplicating the validation here keeps config from + // importing the observability package. + if err := validateOTelHeaders(c.OTel.Headers); err != nil { + return fmt.Errorf("otel.headers: %w", err) + } } // Prometheus exposition is independent of OTel — operators can run it diff --git a/internal/config/config_test.go b/internal/config/config_test.go index 974b0f77..3783b9ad 100644 --- a/internal/config/config_test.go +++ b/internal/config/config_test.go @@ -39,6 +39,10 @@ func TestLoad_Defaults(t *testing.T) { assert.True(t, cfg.OTel.Metrics.Enabled) assert.True(t, cfg.OTel.Logs.Enabled) assert.InEpsilon(t, 1.0, cfg.OTel.Logs.SampleRate, 0.0001) + assert.Equal(t, "", cfg.OTel.Headers) + assert.Equal(t, "", cfg.OTel.Traces.Addr) + assert.Equal(t, "", cfg.OTel.Metrics.Addr) + assert.Equal(t, "", cfg.OTel.Logs.Addr) } func TestLoad_FromYAML(t *testing.T) { @@ -312,6 +316,65 @@ func TestValidate_RejectsEmptyOTelAddrWhenEnabled(t *testing.T) { assert.Contains(t, err.Error(), "otel.addr") } +func TestValidate_RejectsMalformedHeaders(t *testing.T) { + t.Parallel() + cases := []struct { + name string + headers string + }{ + {name: "missing equals", headers: "not-a-pair"}, + {name: "empty key", headers: "=value"}, + {name: "mixed valid and invalid", headers: "a=1,broken"}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + cfg := Config{ + Server: Server{Port: 8080}, + ClickHouse: ClickHouse{HTTPScheme: "http"}, + Schema: Schema{RefreshInterval: 60}, + OTel: OTel{ + Enabled: true, + Addr: "127.0.0.1:4317", + Headers: tc.headers, + }, + } + err := cfg.Validate() + require.Error(t, err) + assert.Contains(t, err.Error(), "otel.headers") + }) + } +} + +func TestValidate_AcceptsValidHeaders(t *testing.T) { + t.Parallel() + cfg := Config{ + Server: Server{Port: 8080}, + ClickHouse: ClickHouse{HTTPScheme: "http"}, + Schema: Schema{RefreshInterval: 60}, + OTel: OTel{ + Enabled: true, + Addr: "127.0.0.1:4317", + Headers: "authorization=Basic dXNlcjpwYXNz==,x-tenant-id=acme", + }, + } + assert.NoError(t, cfg.Validate()) +} + +func TestValidate_HeadersIgnoredWhenOTelDisabled(t *testing.T) { + t.Parallel() + // Iterating on yaml shouldn't fail boot for headers in a disabled block. + cfg := Config{ + Server: Server{Port: 8080}, + ClickHouse: ClickHouse{HTTPScheme: "http"}, + Schema: Schema{RefreshInterval: 60}, + OTel: OTel{ + Enabled: false, + Headers: "not-a-pair", + }, + } + assert.NoError(t, cfg.Validate()) +} + func TestValidate_SampleRatesIgnoredWhenSignalDisabled(t *testing.T) { t.Parallel() // Same idea one level down — when the master switch is on but the individual diff --git a/internal/observability/endpoint.go b/internal/observability/endpoint.go new file mode 100644 index 00000000..89de56b1 --- /dev/null +++ b/internal/observability/endpoint.go @@ -0,0 +1,76 @@ +package observability + +import ( + "crypto/tls" + "fmt" + "strings" +) + +// tlsConfigOrDefault returns the supplied config when non-nil. The OTel SDK's +// credentials.NewTLS expects a non-nil *tls.Config; passing nil panics. An +// empty &tls.Config{} delegates to system defaults (system root CAs, ALPN +// negotiation), which is what production wants for cloud OTLP endpoints. +func tlsConfigOrDefault(c *tls.Config) *tls.Config { + if c != nil { + return c + } + return &tls.Config{} +} + +// ParseEndpoint splits an OTLP endpoint string into the gRPC dial host and a +// useTLS flag. The OpenTelemetry SDK env-var convention is honored: an +// `https://` prefix selects TLS, while `http://` or a bare `host:port` stays +// plaintext (backward-compat with the prior WithInsecure() default). +// +// A URL path component is tolerated and stripped — gRPC routes by service name +// and ignores the path, so `https://otlp-gateway.example.com/otlp` and +// `https://otlp-gateway.example.com` dial the same way. +func ParseEndpoint(addr string) (host string, useTLS bool) { + switch { + case strings.HasPrefix(addr, "https://"): + useTLS = true + host = strings.TrimPrefix(addr, "https://") + case strings.HasPrefix(addr, "http://"): + host = strings.TrimPrefix(addr, "http://") + default: + host = addr + } + if i := strings.IndexByte(host, '/'); i >= 0 { + host = host[:i] + } + return host, useTLS +} + +// ParseOTelHeaders parses the OpenTelemetry-spec headers env-var format +// (`OTEL_EXPORTER_OTLP_HEADERS`) — comma-separated `key=value` pairs — into a +// map. Whitespace around the key and value is trimmed. Only the first `=` per +// segment splits key from value, so base64 trailing `=` in an Authorization +// header round-trips unchanged. An empty input yields an empty map. +// +// Returns an error (rather than silently dropping the segment) for malformed +// entries so Validate() can fail loud at boot rather than letting a typo +// silently disable auth in production. +func ParseOTelHeaders(s string) (map[string]string, error) { + s = strings.TrimSpace(s) + if s == "" { + return map[string]string{}, nil + } + out := map[string]string{} + for _, seg := range strings.Split(s, ",") { + seg = strings.TrimSpace(seg) + if seg == "" { + continue + } + i := strings.IndexByte(seg, '=') + if i < 0 { + return nil, fmt.Errorf("header segment %q missing '='", seg) + } + key := strings.TrimSpace(seg[:i]) + val := strings.TrimSpace(seg[i+1:]) + if key == "" { + return nil, fmt.Errorf("header segment %q has empty key", seg) + } + out[key] = val + } + return out, nil +} diff --git a/internal/observability/endpoint_test.go b/internal/observability/endpoint_test.go new file mode 100644 index 00000000..b59da76c --- /dev/null +++ b/internal/observability/endpoint_test.go @@ -0,0 +1,65 @@ +package observability + +import ( + "reflect" + "testing" +) + +func TestParseEndpoint(t *testing.T) { + cases := []struct { + name string + in string + wantHost string + wantTLS bool + }{ + {name: "bare host port", in: "otlp.example.com:4317", wantHost: "otlp.example.com:4317"}, + {name: "http scheme", in: "http://otlp.example.com:4317", wantHost: "otlp.example.com:4317"}, + {name: "https scheme", in: "https://otlp.example.com:443", wantHost: "otlp.example.com:443", wantTLS: true}, + {name: "https with path", in: "https://otlp-gateway.example.com/otlp", wantHost: "otlp-gateway.example.com", wantTLS: true}, + {name: "https with port and path", in: "https://otlp.example.com:443/v1/traces", wantHost: "otlp.example.com:443", wantTLS: true}, + {name: "empty", in: "", wantHost: ""}, + {name: "ipv4 plaintext", in: "127.0.0.1:4317", wantHost: "127.0.0.1:4317"}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + gotHost, gotTLS := ParseEndpoint(tc.in) + if gotHost != tc.wantHost || gotTLS != tc.wantTLS { + t.Fatalf("ParseEndpoint(%q) = (%q, %v); want (%q, %v)", tc.in, gotHost, gotTLS, tc.wantHost, tc.wantTLS) + } + }) + } +} + +func TestParseOTelHeaders(t *testing.T) { + cases := []struct { + name string + in string + want map[string]string + wantErr bool + }{ + {name: "empty", in: "", want: map[string]string{}}, + {name: "whitespace only", in: " ", want: map[string]string{}}, + {name: "one pair", in: "x-honeycomb-team=abc123", want: map[string]string{"x-honeycomb-team": "abc123"}}, + {name: "two pairs", in: "a=1,b=2", want: map[string]string{"a": "1", "b": "2"}}, + {name: "whitespace around segments", in: " a = 1 , b = 2 ", want: map[string]string{"a": "1", "b": "2"}}, + {name: "value contains equals", in: "authorization=Basic dXNlcjpwYXNz==", want: map[string]string{"authorization": "Basic dXNlcjpwYXNz=="}}, + {name: "trailing comma tolerated", in: "a=1,", want: map[string]string{"a": "1"}}, + {name: "missing equals", in: "not-a-pair", wantErr: true}, + {name: "empty key", in: "=value", wantErr: true}, + {name: "empty value allowed", in: "k=", want: map[string]string{"k": ""}}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + got, err := ParseOTelHeaders(tc.in) + if (err != nil) != tc.wantErr { + t.Fatalf("ParseOTelHeaders(%q) err = %v; wantErr = %v", tc.in, err, tc.wantErr) + } + if tc.wantErr { + return + } + if !reflect.DeepEqual(got, tc.want) { + t.Fatalf("ParseOTelHeaders(%q) = %v; want %v", tc.in, got, tc.want) + } + }) + } +} diff --git a/internal/observability/provider.go b/internal/observability/provider.go index 7ae2da03..2210735c 100644 --- a/internal/observability/provider.go +++ b/internal/observability/provider.go @@ -2,6 +2,7 @@ package observability import ( "context" + "crypto/tls" "errors" "log/slog" "net/http" @@ -24,6 +25,7 @@ import ( "go.opentelemetry.io/otel/sdk/resource" "go.opentelemetry.io/otel/sdk/trace" semconv "go.opentelemetry.io/otel/semconv/v1.24.0" + "google.golang.org/grpc/credentials" ) // runtimeStartOnce gates the `runtime.Start` call in InitProvider so that @@ -42,15 +44,39 @@ var runtimeStartOnce sync.Once // set — the underlying OTel MeterProvider is the shared substrate. When // PrometheusEnabled is true InitProvider returns a non-nil promHandler. // -// Endpoint is the OTLP gRPC target. It is dialed only by the OTLP exporters -// (traces / metrics-OTLP / logs); Prometheus-only operation leaves it unused. +// Endpoint is the OTLP gRPC target consulted by every enabled OTLP exporter +// unless that signal sets its own TracesEndpoint / MetricsEndpoint / +// LogsEndpoint override (empty means inherit). Each endpoint is parsed via +// ParseEndpoint, so `https://` selects TLS while bare `host:port` or `http://` +// stays plaintext for backward compatibility. +// +// Headers (set, key=value pairs already parsed by config) is applied to every +// OTLP exporter — the standard knob for auth against cloud endpoints. +// +// TLSConfig is a test-only escape hatch: when non-nil it overrides the default +// system-roots TLS config for `https://` endpoints. Production code leaves it +// nil; only the FakeOTLPTLS-driven integration tests populate it (with +// InsecureSkipVerify against an ephemeral self-signed cert). type ProviderConfig struct { Endpoint string + Headers map[string]string + TracesEndpoint string TracesEnabled bool TracesSampleRate float64 + MetricsEndpoint string MetricsEnabled bool PrometheusEnabled bool + LogsEndpoint string LogsEnabled bool + TLSConfig *tls.Config +} + +// pickEndpoint returns override if set, otherwise fallback. +func pickEndpoint(override, fallback string) string { + if override != "" { + return override + } + return fallback } // InitProvider sets up the OpenTelemetry pipeline, registering only the @@ -115,10 +141,17 @@ func InitProvider(ctx context.Context, serviceName string, cfg ProviderConfig) ( ) if cfg.TracesEnabled { - traceExporter, err := otlptracegrpc.New(ctx, - otlptracegrpc.WithEndpoint(cfg.Endpoint), - otlptracegrpc.WithInsecure(), - ) + host, useTLS := ParseEndpoint(pickEndpoint(cfg.TracesEndpoint, cfg.Endpoint)) + opts := []otlptracegrpc.Option{otlptracegrpc.WithEndpoint(host)} + if useTLS { + opts = append(opts, otlptracegrpc.WithTLSCredentials(credentials.NewTLS(tlsConfigOrDefault(cfg.TLSConfig)))) + } else { + opts = append(opts, otlptracegrpc.WithInsecure()) + } + if len(cfg.Headers) > 0 { + opts = append(opts, otlptracegrpc.WithHeaders(cfg.Headers)) + } + traceExporter, err := otlptracegrpc.New(ctx, opts...) if err != nil { handleErr(err) return shutdown, nil, err @@ -138,10 +171,17 @@ func InitProvider(ctx context.Context, serviceName string, cfg ProviderConfig) ( readers := []metric.Reader{} if cfg.MetricsEnabled { - metricExporter, err := otlpmetricgrpc.New(ctx, - otlpmetricgrpc.WithEndpoint(cfg.Endpoint), - otlpmetricgrpc.WithInsecure(), - ) + host, useTLS := ParseEndpoint(pickEndpoint(cfg.MetricsEndpoint, cfg.Endpoint)) + opts := []otlpmetricgrpc.Option{otlpmetricgrpc.WithEndpoint(host)} + if useTLS { + opts = append(opts, otlpmetricgrpc.WithTLSCredentials(credentials.NewTLS(tlsConfigOrDefault(cfg.TLSConfig)))) + } else { + opts = append(opts, otlpmetricgrpc.WithInsecure()) + } + if len(cfg.Headers) > 0 { + opts = append(opts, otlpmetricgrpc.WithHeaders(cfg.Headers)) + } + metricExporter, err := otlpmetricgrpc.New(ctx, opts...) if err != nil { handleErr(err) return shutdown, nil, err @@ -195,10 +235,17 @@ func InitProvider(ctx context.Context, serviceName string, cfg ProviderConfig) ( } if cfg.LogsEnabled { - logExporter, err := otlploggrpc.New(ctx, - otlploggrpc.WithEndpoint(cfg.Endpoint), - otlploggrpc.WithInsecure(), - ) + host, useTLS := ParseEndpoint(pickEndpoint(cfg.LogsEndpoint, cfg.Endpoint)) + opts := []otlploggrpc.Option{otlploggrpc.WithEndpoint(host)} + if useTLS { + opts = append(opts, otlploggrpc.WithTLSCredentials(credentials.NewTLS(tlsConfigOrDefault(cfg.TLSConfig)))) + } else { + opts = append(opts, otlploggrpc.WithInsecure()) + } + if len(cfg.Headers) > 0 { + opts = append(opts, otlploggrpc.WithHeaders(cfg.Headers)) + } + logExporter, err := otlploggrpc.New(ctx, opts...) if err != nil { handleErr(err) return shutdown, nil, err diff --git a/internal/testutil/otlp.go b/internal/testutil/otlp.go index 1fea94bd..518018df 100644 --- a/internal/testutil/otlp.go +++ b/internal/testutil/otlp.go @@ -15,15 +15,28 @@ // // Always call shutdown before asserting counts — the OTel SDK batches // exports and only drains on shutdown (or after the batch timeout). +// +// For TLS verification, use NewFakeOTLPTLS which mints an ephemeral self-signed +// cert and exposes TLSConfig() for a matching client-side config. package testutil import ( "context" + "crypto/ecdsa" + "crypto/elliptic" + "crypto/rand" + "crypto/tls" + "crypto/x509" + "crypto/x509/pkix" + "math/big" "net" "sync" "testing" + "time" "google.golang.org/grpc" + "google.golang.org/grpc/credentials" + "google.golang.org/grpc/metadata" collogspb "go.opentelemetry.io/proto/otlp/collector/logs/v1" colmetricspb "go.opentelemetry.io/proto/otlp/collector/metrics/v1" @@ -37,19 +50,51 @@ import ( // log Export RPCs and captures every received payload. Cleanup is registered // on the *testing.T automatically. type FakeOTLP struct { - addr string - server *grpc.Server + addr string + server *grpc.Server + tlsConfig *tls.Config // non-nil only when constructed via NewFakeOTLPTLS mu sync.Mutex traces []*tracepb.ResourceSpans metrics []*metricspb.ResourceMetrics logs []*logspb.ResourceLogs + + // Captured gRPC request metadata, indexed by signal. Each Export call + // appends an entry. Tests assert on auth/header propagation through these. + traceHeaders []metadata.MD + metricHeaders []metadata.MD + logHeaders []metadata.MD } // NewFakeOTLP binds the receiver to 127.0.0.1 on a random port and starts -// serving. The server is stopped automatically when the test ends. +// serving plaintext gRPC. The server is stopped automatically when the test +// ends. func NewFakeOTLP(t *testing.T) *FakeOTLP { t.Helper() + return newFakeOTLP(t, nil) +} + +// NewFakeOTLPTLS is the TLS variant: an ephemeral self-signed cert (SAN +// 127.0.0.1) is generated, and the server listens with that cert. The matching +// client config is available via TLSConfig() — wire it into ProviderConfig so +// the OTel exporters trust the cert. Production code never sets ProviderConfig.TLSConfig; +// only this test path does. +func NewFakeOTLPTLS(t *testing.T) *FakeOTLP { + t.Helper() + + cert, clientCfg := ephemeralTLSPair(t) + serverCfg := &tls.Config{Certificates: []tls.Certificate{cert}} + + return newFakeOTLP(t, &fakeOTLPTLS{server: serverCfg, client: clientCfg}) +} + +type fakeOTLPTLS struct { + server *tls.Config + client *tls.Config +} + +func newFakeOTLP(t *testing.T, tlsCfg *fakeOTLPTLS) *FakeOTLP { + t.Helper() var lc net.ListenConfig lis, err := lc.Listen(t.Context(), "tcp", "127.0.0.1:0") @@ -57,10 +102,14 @@ func NewFakeOTLP(t *testing.T) *FakeOTLP { t.Fatalf("FakeOTLP listen: %v", err) } - r := &FakeOTLP{ - addr: lis.Addr().String(), - server: grpc.NewServer(), + var serverOpts []grpc.ServerOption + r := &FakeOTLP{addr: lis.Addr().String()} + if tlsCfg != nil { + serverOpts = append(serverOpts, grpc.Creds(credentials.NewTLS(tlsCfg.server))) + r.tlsConfig = tlsCfg.client } + r.server = grpc.NewServer(serverOpts...) + coltracepb.RegisterTraceServiceServer(r.server, &fakeTraceServer{parent: r}) colmetricspb.RegisterMetricsServiceServer(r.server, &fakeMetricsServer{parent: r}) collogspb.RegisterLogsServiceServer(r.server, &fakeLogsServer{parent: r}) @@ -76,10 +125,55 @@ func NewFakeOTLP(t *testing.T) *FakeOTLP { return r } +// ephemeralTLSPair mints a one-shot ECDSA self-signed cert valid for 127.0.0.1 +// and returns it along with a client tls.Config that trusts only this cert. +func ephemeralTLSPair(t *testing.T) (tls.Certificate, *tls.Config) { + t.Helper() + + priv, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader) + if err != nil { + t.Fatalf("FakeOTLPTLS: generate key: %v", err) + } + serial, err := rand.Int(rand.Reader, big.NewInt(1<<62)) + if err != nil { + t.Fatalf("FakeOTLPTLS: serial: %v", err) + } + tmpl := x509.Certificate{ + SerialNumber: serial, + Subject: pkix.Name{CommonName: "FakeOTLP"}, + NotBefore: time.Now().Add(-time.Minute), + NotAfter: time.Now().Add(time.Hour), + KeyUsage: x509.KeyUsageDigitalSignature, + ExtKeyUsage: []x509.ExtKeyUsage{x509.ExtKeyUsageServerAuth}, + IPAddresses: []net.IP{net.ParseIP("127.0.0.1")}, + DNSNames: []string{"localhost"}, + } + der, err := x509.CreateCertificate(rand.Reader, &tmpl, &tmpl, &priv.PublicKey, priv) + if err != nil { + t.Fatalf("FakeOTLPTLS: create cert: %v", err) + } + cert := tls.Certificate{ + Certificate: [][]byte{der}, + PrivateKey: priv, + } + parsed, err := x509.ParseCertificate(der) + if err != nil { + t.Fatalf("FakeOTLPTLS: parse cert: %v", err) + } + pool := x509.NewCertPool() + pool.AddCert(parsed) + return cert, &tls.Config{RootCAs: pool, ServerName: "127.0.0.1"} +} + // Addr returns the listener address (e.g. "127.0.0.1:42891") suitable for // passing to observability.ProviderConfig.Endpoint. func (r *FakeOTLP) Addr() string { return r.addr } +// TLSConfig returns the client-side tls.Config that trusts this server's +// ephemeral cert. Returns nil when the server was constructed via the plaintext +// NewFakeOTLP. +func (r *FakeOTLP) TLSConfig() *tls.Config { return r.tlsConfig } + // SpanCount returns the total number of spans received across all RPCs. // Spans are flattened across resource and scope groupings. func (r *FakeOTLP) SpanCount() int { @@ -139,6 +233,27 @@ func (r *FakeOTLP) LogCountAtLevel(minSeverity int32) int { return n } +// LastTraceHeaders returns the gRPC metadata captured from the most recent +// trace Export RPC, or nil if none. +func (r *FakeOTLP) LastTraceHeaders() metadata.MD { return lastMD(&r.mu, r.traceHeaders) } + +// LastMetricHeaders returns the gRPC metadata captured from the most recent +// metric Export RPC, or nil if none. +func (r *FakeOTLP) LastMetricHeaders() metadata.MD { return lastMD(&r.mu, r.metricHeaders) } + +// LastLogHeaders returns the gRPC metadata captured from the most recent log +// Export RPC, or nil if none. +func (r *FakeOTLP) LastLogHeaders() metadata.MD { return lastMD(&r.mu, r.logHeaders) } + +func lastMD(mu *sync.Mutex, slice []metadata.MD) metadata.MD { + mu.Lock() + defer mu.Unlock() + if len(slice) == 0 { + return nil + } + return slice[len(slice)-1] +} + // Reset clears all captured payloads. Useful between test phases. func (r *FakeOTLP) Reset() { r.mu.Lock() @@ -146,6 +261,9 @@ func (r *FakeOTLP) Reset() { r.traces = nil r.metrics = nil r.logs = nil + r.traceHeaders = nil + r.metricHeaders = nil + r.logHeaders = nil } type fakeTraceServer struct { @@ -153,9 +271,11 @@ type fakeTraceServer struct { parent *FakeOTLP } -func (s *fakeTraceServer) Export(_ context.Context, req *coltracepb.ExportTraceServiceRequest) (*coltracepb.ExportTraceServiceResponse, error) { +func (s *fakeTraceServer) Export(ctx context.Context, req *coltracepb.ExportTraceServiceRequest) (*coltracepb.ExportTraceServiceResponse, error) { + md, _ := metadata.FromIncomingContext(ctx) s.parent.mu.Lock() s.parent.traces = append(s.parent.traces, req.GetResourceSpans()...) + s.parent.traceHeaders = append(s.parent.traceHeaders, md.Copy()) s.parent.mu.Unlock() return &coltracepb.ExportTraceServiceResponse{}, nil } @@ -165,9 +285,11 @@ type fakeMetricsServer struct { parent *FakeOTLP } -func (s *fakeMetricsServer) Export(_ context.Context, req *colmetricspb.ExportMetricsServiceRequest) (*colmetricspb.ExportMetricsServiceResponse, error) { +func (s *fakeMetricsServer) Export(ctx context.Context, req *colmetricspb.ExportMetricsServiceRequest) (*colmetricspb.ExportMetricsServiceResponse, error) { + md, _ := metadata.FromIncomingContext(ctx) s.parent.mu.Lock() s.parent.metrics = append(s.parent.metrics, req.GetResourceMetrics()...) + s.parent.metricHeaders = append(s.parent.metricHeaders, md.Copy()) s.parent.mu.Unlock() return &colmetricspb.ExportMetricsServiceResponse{}, nil } @@ -177,9 +299,11 @@ type fakeLogsServer struct { parent *FakeOTLP } -func (s *fakeLogsServer) Export(_ context.Context, req *collogspb.ExportLogsServiceRequest) (*collogspb.ExportLogsServiceResponse, error) { +func (s *fakeLogsServer) Export(ctx context.Context, req *collogspb.ExportLogsServiceRequest) (*collogspb.ExportLogsServiceResponse, error) { + md, _ := metadata.FromIncomingContext(ctx) s.parent.mu.Lock() s.parent.logs = append(s.parent.logs, req.GetResourceLogs()...) + s.parent.logHeaders = append(s.parent.logHeaders, md.Copy()) s.parent.mu.Unlock() return &collogspb.ExportLogsServiceResponse{}, nil } diff --git a/tests/integration/otel_test.go b/tests/integration/otel_test.go index aee9843b..77644a6a 100644 --- a/tests/integration/otel_test.go +++ b/tests/integration/otel_test.go @@ -313,3 +313,121 @@ func TestOTel_UnreachableEndpoint_DoesNotBlockStartupOrEmits(t *testing.T) { _ = shutdown(drainCtx) }() } + +// TestOTel_TLSPath_Traces locks in the https:// → TLS dial path. The fake +// receiver listens with an ephemeral self-signed cert; the exporter is given +// the matching client tls.Config via ProviderConfig.TLSConfig. If TLS wiring +// regresses (e.g. someone re-adds WithInsecure unconditionally), the dial +// will TLS-handshake against a plaintext server and the export drops. +func TestOTel_TLSPath_Traces(t *testing.T) { + guardOTelGlobals(t) + r := testutil.NewFakeOTLPTLS(t) + + shutdown, _ := initAndShutdown(t, observability.ProviderConfig{ + Endpoint: "https://" + r.Addr(), + TLSConfig: r.TLSConfig(), + TracesEnabled: true, + TracesSampleRate: 1.0, + }) + + _, span := otel.Tracer("test").Start(context.Background(), "tls-op") + span.End() + + drainCtx, drainCancel := context.WithTimeout(context.Background(), 5*time.Second) + defer drainCancel() + require.NoError(t, shutdown(drainCtx)) + + assert.Equal(t, 1, r.SpanCount(), "TLS path must deliver the span end-to-end") +} + +// TestOTel_Headers_AppliedToAllSignals verifies that ProviderConfig.Headers +// propagates as gRPC metadata on every OTLP exporter (traces, metrics, logs). +// Direct-to-cloud auth depends on this — Honeycomb/Grafana Cloud both +// authenticate per-RPC via a header, so a single missing exporter would 401 +// silently for that signal. +func TestOTel_Headers_AppliedToAllSignals(t *testing.T) { + guardOTelGlobals(t) + r := testutil.NewFakeOTLP(t) + + headers := map[string]string{ + "authorization": "Bearer test-token", + "x-honeycomb-team": "abc123", + } + shutdown, _ := initAndShutdown(t, observability.ProviderConfig{ + Endpoint: r.Addr(), + Headers: headers, + TracesEnabled: true, + TracesSampleRate: 1.0, + MetricsEnabled: true, + LogsEnabled: true, + }) + + // Emit one of each signal. + _, span := otel.Tracer("test").Start(context.Background(), "auth-op") + span.End() + + counter, err := otel.GetMeterProvider().Meter("test").Int64Counter("hdr_counter") + require.NoError(t, err) + counter.Add(context.Background(), 1) + + lvl := &slog.LevelVar{} + lvl.Set(slog.LevelInfo) + logger := observability.NewLogger("wavehouse-test", lvl, true, 1.0) + logger.Info("auth-log") + + drainCtx, drainCancel := context.WithTimeout(context.Background(), 5*time.Second) + defer drainCancel() + require.NoError(t, shutdown(drainCtx)) + + // gRPC metadata keys are lowercased on the wire — assert in lowercase. + for _, sig := range []struct { + name string + md func() []string + }{ + {"traces", func() []string { return r.LastTraceHeaders().Get("authorization") }}, + {"metrics", func() []string { return r.LastMetricHeaders().Get("authorization") }}, + {"logs", func() []string { return r.LastLogHeaders().Get("authorization") }}, + } { + t.Run(sig.name, func(t *testing.T) { + vals := sig.md() + require.NotEmpty(t, vals, "%s exporter dropped the authorization header", sig.name) + assert.Equal(t, "Bearer test-token", vals[0]) + }) + } + // Spot-check the second header on at least one signal — same map flows + // through all three so one check confirms multi-header support. + assert.Equal(t, []string{"abc123"}, r.LastTraceHeaders().Get("x-honeycomb-team")) +} + +// TestOTel_PerSignalEndpoint_Override sends traces to one receiver and metrics +// to another by setting TracesEndpoint on top of a default Endpoint. Grafana +// Cloud's distinct gateway hosts per signal are the headline use case. +func TestOTel_PerSignalEndpoint_Override(t *testing.T) { + guardOTelGlobals(t) + rDefault := testutil.NewFakeOTLP(t) + rTraces := testutil.NewFakeOTLP(t) + + shutdown, _ := initAndShutdown(t, observability.ProviderConfig{ + Endpoint: rDefault.Addr(), // metrics + logs land here + TracesEndpoint: rTraces.Addr(), // traces override + TracesEnabled: true, + TracesSampleRate: 1.0, + MetricsEnabled: true, + }) + + _, span := otel.Tracer("test").Start(context.Background(), "split-op") + span.End() + + counter, err := otel.GetMeterProvider().Meter("test").Int64Counter("split_counter") + require.NoError(t, err) + counter.Add(context.Background(), 1) + + drainCtx, drainCancel := context.WithTimeout(context.Background(), 5*time.Second) + defer drainCancel() + require.NoError(t, shutdown(drainCtx)) + + assert.Equal(t, 1, rTraces.SpanCount(), "trace endpoint should have received the span") + assert.Zero(t, rDefault.SpanCount(), "default endpoint must NOT receive traces when overridden") + assert.GreaterOrEqual(t, rDefault.MetricCount(), 1, "default endpoint should receive metrics (no override)") + assert.Zero(t, rTraces.MetricCount(), "trace endpoint should NOT receive metrics") +} From 607db8c6d0886f28185b9e3f080a0f18ee84b0e2 Mon Sep 17 00:00:00 2001 From: Jack Woods Date: Tue, 12 May 2026 23:33:14 -0400 Subject: [PATCH 02/25] =?UTF-8?q?chore(deploy):=20local=20compose=20stack?= =?UTF-8?q?=20=E2=80=94=20fix=20SigNoz,=20wire=20WaveHouse=20=E2=86=92=20S?= =?UTF-8?q?igNoz?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Rebuild deployments/signoz/ from the upstream SigNoz v0.122.0 deploy (single node). The old hand-rolled compose pinned :latest everywhere and broke: the current signoz-otel-collector migrate CLI rejects an empty --clickhouse-cluster (emits a bare `ON CLUSTER ` → ClickHouse code 62), and signoz/query-service has been superseded by the consolidated signoz/signoz image. - deployments/signoz/: signoz/signoz:v0.122.0 (UI/API on :3301) + signoz-otel-collector:v0.144.3 (collector + dedicated schema migrator, driven by SIGNOZ_OTEL_COLLECTOR_CLICKHOUSE_CLUSTER=cluster) + clickhouse-server:25.5.6 + zookeeper:3.7.1 + one-shot init-clickhouse; vendored clickhouse/{config,users,cluster,custom-function}.xml and otel-collector-opamp-config.yaml; dropped the now-obsolete .env.example - deployments/compose/standalone.signoz.yaml: override that wires the containerized standalone WaveHouse into SigNoz (WH_OTEL_ADDR=host.docker.internal:4317), caps WH_MQ_MAX_BYTES_GB for small dev disks, and gates WaveHouse on a ClickHouse healthcheck - WHissues.md: log the issues hit along the way (SigNoz compose rot, standalone.yaml's 50 GB JetStream default, missing CH readiness gate, the clickhouse service-name collision when dual-homing networks) - docs/deployment.md: refresh the SigNoz section Local dev tooling only — intentionally kept off the observability PR branch. Co-Authored-By: Claude Opus 4.7 (1M context) --- WHissues.md | 140 ++ deployments/compose/standalone.signoz.yaml | 47 + deployments/signoz/.env.example | 5 - deployments/signoz/clickhouse/cluster.xml | 75 ++ deployments/signoz/clickhouse/config.xml | 1142 +++++++++++++++++ .../signoz/clickhouse/custom-function.xml | 21 + .../signoz/clickhouse/user_scripts/.gitkeep | 0 deployments/signoz/clickhouse/users.xml | 123 ++ deployments/signoz/docker-compose.yaml | 222 +++- deployments/signoz/otel-collector-config.yaml | 10 +- .../signoz/otel-collector-opamp-config.yaml | 1 + docs/src/content/docs/deployment.md | 6 +- 12 files changed, 1716 insertions(+), 76 deletions(-) create mode 100644 WHissues.md create mode 100644 deployments/compose/standalone.signoz.yaml delete mode 100644 deployments/signoz/.env.example create mode 100644 deployments/signoz/clickhouse/cluster.xml create mode 100644 deployments/signoz/clickhouse/config.xml create mode 100644 deployments/signoz/clickhouse/custom-function.xml create mode 100644 deployments/signoz/clickhouse/user_scripts/.gitkeep create mode 100644 deployments/signoz/clickhouse/users.xml create mode 100644 deployments/signoz/otel-collector-opamp-config.yaml diff --git a/WHissues.md b/WHissues.md new file mode 100644 index 00000000..8a86588a --- /dev/null +++ b/WHissues.md @@ -0,0 +1,140 @@ +# WaveHouse — Known Issues + +Running log of issues found while deploying/operating WaveHouse. Newest first. + +--- + +## 1. SigNoz compose stack (`deployments/signoz/docker-compose.yaml`) fails to start — RESOLVED + +**Found:** 2026-05-12, bringing the bundled SigNoz stack up locally on OrbStack. +**Fixed:** 2026-05-12 — `deployments/signoz/` rebuilt from the upstream SigNoz +`deploy/docker/docker-compose.yaml` (SigNoz `v0.122.0` / `signoz-otel-collector +v0.144.3`), trimmed to a single node. + +**Symptom (before):** `docker compose -f deployments/signoz/docker-compose.yaml up -d` +→ `signoz-migrator` exits 1; consequently `signoz` (query-service) and +`signoz-otel-collector` never start (both `depends_on: migrator: +service_completed_successfully`). Only `signoz-clickhouse` stays up. + +Migrator log: + +``` +migrate/bootstrap.go:119 Creating databases +Error: failed to bootstrap store for migrations: code: 62, message: Syntax error: failed at position 42 (end of query): . Expected one of: identifier, string literal +``` + +**Cause:** the old compose file pinned `:latest` on every SigNoz image. The current +`signoz/signoz-otel-collector:latest` migrate CLI defaults `--clickhouse-cluster` +to `cluster` and `--clickhouse-replication` to `true`; the compose passed +`--clickhouse-cluster=""` (empty). An empty cluster name made the bootstrap step +emit a `... ON CLUSTER ` clause with nothing after it → ClickHouse `code: 62` +syntax error. Secondary problem: `signoz/query-service:latest` was a ~14-month-old +build (2025-03) while `signoz/signoz-otel-collector:latest` is from 2026-05 — those +are no longer a compatible pair (`signoz/query-service` has been superseded by the +consolidated `signoz/signoz` image). + +**Fix applied:** rebuilt `deployments/signoz/` to mirror upstream's current single-node +layout, with everything version-pinned: +- `signoz/signoz:v0.122.0` (consolidated query + UI; replaces `signoz/query-service`), + published on host `3301` (→ container `8080`) +- `signoz/signoz-otel-collector:v0.144.3` (OTLP gRPC `4317`, HTTP `4318`) — runs + `migrate sync check` then the collector, with `--manager-config` pointing at the + bundled `otel-collector-opamp-config.yaml` +- `signoz-telemetrystore-migrator` (same collector image) — `migrate bootstrap && sync + up && async up`, driven by `SIGNOZ_OTEL_COLLECTOR_CLICKHOUSE_CLUSTER=cluster` / + `…_REPLICATION=true` env vars (the cluster `ON CLUSTER cluster` DDL now resolves) +- `clickhouse/clickhouse-server:25.5.6` with `clickhouse/{config,users,cluster,custom-function}.xml` + vendored under `deployments/signoz/clickhouse/` — `cluster.xml` defines a 1-shard / + 1-replica cluster named `cluster` and points at `zookeeper-1` +- `signoz/zookeeper:3.7.1` — needed for the `ON CLUSTER` / replicated tables +- `signoz-init-clickhouse` (one-shot) — downloads the `histogramQuantile` UDF binary + into a shared `signoz-clickhouse-user-scripts` volume +- removed the old `deployments/signoz/.env.example` (the `SIGNOZ_CH_USER` / + `SIGNOZ_CH_PASSWORD` knobs no longer exist; ClickHouse uses `CLICKHOUSE_SKIP_USER_SETUP=1` + + the vendored `users.xml`, which has the `default` user with an empty password) + +WaveHouse is wired in via `deployments/compose/standalone.signoz.yaml` +(`WH_OTEL_ENABLED=true`, `WH_OTEL_ADDR=host.docker.internal:4317`); verified that +spans from `service.name=wavehouse` land in `signoz_traces.distributed_signoz_index_v3`. + +**Note:** on first start the collector logs an OpAMP error +(`cannot create agent without orgId` / `Server returned an error response`) and the +WaveHouse OTLP exporter logs a few `error reading server preface: EOF` lines while the +collector finishes restarting — both are transient. The OpAMP error clears once you +create the first user/org in the SigNoz UI; the EOF lines stop once the collector is +steady (data flows regardless). + +**Status:** resolved. + +--- + +## 2. `deployments/compose/standalone.yaml` quick-start fails on a default-sized Docker/OrbStack VM + +**Found:** 2026-05-12. + +**Symptom:** `docker compose -f deployments/compose/standalone.yaml up -d --build` → +the `wavehouse` container exits 1 on boot: + +``` +mq init failed error="create stream: nats: API error: code=500 err_code=10047 description=insufficient storage resources available" path=/app/data/nats +``` + +**Cause:** `standalone.yaml` does not set `WH_MQ_MAX_BYTES_GB`, so the embedded +JetStream stream uses the 50 GB default (`config.go`: `MaxBytesGB env-default:"50"`). +A stock OrbStack VM disk is ~28 GB; JetStream caps usable file storage well below +that (~16 GB observed), so creating a 50 GB stream fails. WaveHouse treats the MQ +init failure as fatal. + +**Workaround / fix:** set `WH_MQ_MAX_BYTES_GB` to something that fits in +`standalone.yaml` (the e2e compose uses `1`), or document the disk requirement in +`docs/deployment.md`. Locally this is handled by +`deployments/compose/standalone.signoz.yaml` (sets it to `2`). + +--- + +## 3. `standalone.yaml` doesn't gate WaveHouse on ClickHouse readiness + +**Found:** 2026-05-12. + +**Symptom:** on a cold `up`, the `wavehouse` container exits 1: + +``` +schema discovery failed on boot error="query system.columns: dial tcp :9000: connect: connection refused" +``` + +**Cause:** `standalone.yaml` uses `depends_on: [clickhouse]`, which only waits for +the ClickHouse container to *start*, not to accept connections. WaveHouse runs +schema discovery immediately and treats a failure on boot as fatal (`main.go` +`return 1`), so it races a still-initializing ClickHouse and loses. +`deployments/compose/dependencies.yaml` and `tests/e2e/compose.yaml` both add a +ClickHouse healthcheck and `depends_on: { clickhouse: { condition: +service_healthy } }`; `standalone.yaml` should do the same. + +**Workaround / fix:** add the healthcheck + `service_healthy` condition to +`standalone.yaml` (done in `standalone.signoz.yaml` for the local stack), or restart +the `wavehouse` container after ClickHouse is up. + +--- + +## 4. Gotcha: attaching WaveHouse to the SigNoz network breaks ClickHouse name resolution + +**Found:** 2026-05-12, while wiring WaveHouse's OTLP export into the SigNoz stack. + +**Symptom:** if you put the WaveHouse container on the SigNoz compose network so it +can reach `signoz-otel-collector:4317` by name, schema discovery fails with: + +``` +query system.columns: code: 516, message: default: Authentication failed: password is incorrect, or there is no user with such name. +``` + +**Cause:** `deployments/compose/standalone.yaml` and +`deployments/signoz/docker-compose.yaml` both define a service named `clickhouse`. +On a container attached to both networks, the unqualified hostname `clickhouse` is +ambiguous, and WaveHouse can end up dialing SigNoz's ClickHouse — which is +configured with a non-empty password (`SIGNOZ_CH_PASSWORD`, default `password`), +unlike WaveHouse's own ClickHouse (empty password) — so auth fails. + +**Workaround / fix:** don't dual-home the WaveHouse container. Reach the collector +via `host.docker.internal:4317` (the SigNoz collector publishes `4317` on the host) — +this is what `standalone.signoz.yaml` does. Alternatively, rename one of the +`clickhouse` services so the names don't collide. diff --git a/deployments/compose/standalone.signoz.yaml b/deployments/compose/standalone.signoz.yaml new file mode 100644 index 00000000..e5551f0a --- /dev/null +++ b/deployments/compose/standalone.signoz.yaml @@ -0,0 +1,47 @@ +# Override for standalone.yaml that points WaveHouse's OTLP export at the +# SigNoz collector from deployments/signoz/docker-compose.yaml. +# +# The SigNoz collector publishes 4317 on the host, so WaveHouse reaches it +# via host.docker.internal — no shared Docker network. Joining the SigNoz +# stack's network is tempting but breaks ClickHouse resolution: that stack +# also defines a service named `clickhouse`, so the unqualified hostname +# `clickhouse` becomes ambiguous and WaveHouse may dial SigNoz's ClickHouse +# (non-empty password) instead of its own. +# +# Usage (bring the SigNoz stack up first): +# docker compose -f deployments/signoz/docker-compose.yaml up -d +# docker compose -p wavehouse \ +# -f deployments/compose/standalone.yaml \ +# -f deployments/compose/standalone.signoz.yaml up -d --build + +services: + clickhouse: + # standalone.yaml's `depends_on: [clickhouse]` only waits for the + # container to start, not for ClickHouse to accept connections — and + # WaveHouse treats a failed schema-discovery on boot as fatal, so it + # races ClickHouse and exits. Add a readiness probe so the dependency + # below can gate on it. + healthcheck: + test: + [ + "CMD-SHELL", + "wget --no-verbose --tries=1 --spider http://localhost:8123/ping || exit 1", + ] + interval: 2s + timeout: 2s + retries: 15 + + wavehouse: + environment: + WH_OTEL_ENABLED: "true" + WH_OTEL_ADDR: "host.docker.internal:4317" + # standalone.yaml leaves WH_MQ_MAX_BYTES_GB at its 50 GB default; the + # embedded JetStream stream then asks for more disk than the default + # OrbStack VM has free (~16 GB usable), so stream creation fails with + # "insufficient storage resources". Cap it for local use. + WH_MQ_MAX_BYTES_GB: "2" + extra_hosts: + - "host.docker.internal:host-gateway" + depends_on: + clickhouse: + condition: service_healthy diff --git a/deployments/signoz/.env.example b/deployments/signoz/.env.example deleted file mode 100644 index d7180e7e..00000000 --- a/deployments/signoz/.env.example +++ /dev/null @@ -1,5 +0,0 @@ -# SigNoz internal ClickHouse credentials. -# Copy this file to `.env` (gitignored) and override for non-dev use. -# Defaults below match the compose file's fallbacks, so unset works for local dev. -SIGNOZ_CH_USER=default -SIGNOZ_CH_PASSWORD=password diff --git a/deployments/signoz/clickhouse/cluster.xml b/deployments/signoz/clickhouse/cluster.xml new file mode 100644 index 00000000..8b475ffe --- /dev/null +++ b/deployments/signoz/clickhouse/cluster.xml @@ -0,0 +1,75 @@ + + + + + + zookeeper-1 + 2181 + + + + + + + + + + + + + + + + clickhouse + 9000 + + + + + + + + diff --git a/deployments/signoz/clickhouse/config.xml b/deployments/signoz/clickhouse/config.xml new file mode 100644 index 00000000..1965ac3b --- /dev/null +++ b/deployments/signoz/clickhouse/config.xml @@ -0,0 +1,1142 @@ + + + + + + information + + json + + /var/log/clickhouse-server/clickhouse-server.log + /var/log/clickhouse-server/clickhouse-server.err.log + + 1000M + 10 + + + + + + + + + + + + + + + + + + 8123 + + + 9000 + + + 9004 + + + 9005 + + + + + + + + + + + + 9009 + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + 4096 + + + 3 + + + + + false + + + /path/to/ssl_cert_file + /path/to/ssl_key_file + + + false + + + /path/to/ssl_ca_cert_file + + + none + + + 0 + + + -1 + -1 + + + false + + + + + + + + + + + none + true + true + sslv2,sslv3 + true + + + + true + true + sslv2,sslv3 + true + + + + RejectCertificateHandler + + + + + + + + + 100 + + + 0 + + + + 10000 + + + + + + 0.9 + + + 4194304 + + + 0 + + + + + + 8589934592 + + + 5368709120 + + + + 1000 + + + 134217728 + + + 10000 + + + /var/lib/clickhouse/ + + + /var/lib/clickhouse/tmp/ + + + + ` + + + + + + /var/lib/clickhouse/user_files/ + + + + + + + + + + + + + users.xml + + + + /var/lib/clickhouse/access/ + + + + + + + default + + + + + + + + + + + + default + + + + + + + + + true + + + false + + ' | sed -e 's|.*>\(.*\)<.*|\1|') + wget https://github.com/ClickHouse/clickhouse-jdbc-bridge/releases/download/v$PKG_VER/clickhouse-jdbc-bridge_$PKG_VER-1_all.deb + apt install --no-install-recommends -f ./clickhouse-jdbc-bridge_$PKG_VER-1_all.deb + clickhouse-jdbc-bridge & + + * [CentOS/RHEL] + export MVN_URL=https://repo1.maven.org/maven2/ru/yandex/clickhouse/clickhouse-jdbc-bridge + export PKG_VER=$(curl -sL $MVN_URL/maven-metadata.xml | grep '' | sed -e 's|.*>\(.*\)<.*|\1|') + wget https://github.com/ClickHouse/clickhouse-jdbc-bridge/releases/download/v$PKG_VER/clickhouse-jdbc-bridge-$PKG_VER-1.noarch.rpm + yum localinstall -y clickhouse-jdbc-bridge-$PKG_VER-1.noarch.rpm + clickhouse-jdbc-bridge & + + Please refer to https://github.com/ClickHouse/clickhouse-jdbc-bridge#usage for more information. + ]]> + + + + + + + + + + + + + + + 01 + example01-01-1 + + + + + + 3600 + + + + 3600 + + + 60 + + + + + + + + + + /metrics + 9363 + + true + true + true + true + + + + + + system + query_log
+ + toYYYYMM(event_date) + + + + + + 7500 + + + + + system + trace_log
+ + toYYYYMM(event_date) + 7500 + + + + + system + query_thread_log
+ toYYYYMM(event_date) + 7500 + + + + + system + query_views_log
+ toYYYYMM(event_date) + 7500 + + + + + system + part_log
+ toYYYYMM(event_date) + 7500 + + + + + + + system + metric_log
+ 7500 + 1000 + + + + + system + asynchronous_metric_log
+ + 7000 + + + + + + + engine MergeTree + partition by toYYYYMM(finish_date) + order by (finish_date, finish_time_us, trace_id) + + system + opentelemetry_span_log
+ 7500 + + + + + + system + crash_log
+ + + 1000 + + + + + + + + system + processors_profile_log
+ + toYYYYMM(event_date) + 7500 + + + + + + + + + + *_dictionary.xml + + + *function.xml + /var/lib/clickhouse/user_scripts/ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + /clickhouse/task_queue/ddl + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + click_cost + any + + 0 + 3600 + + + 86400 + 60 + + + + max + + 0 + 60 + + + 3600 + 300 + + + 86400 + 3600 + + + + + + /var/lib/clickhouse/format_schemas/ + + + + + hide encrypt/decrypt arguments + ((?:aes_)?(?:encrypt|decrypt)(?:_mysql)?)\s*\(\s*(?:'(?:\\'|.)+'|.*?)\s*\) + + \1(???) + + + + + + + + + + false + + false + + + https://6f33034cfe684dd7a3ab9875e57b1c8d@o388870.ingest.sentry.io/5226277 + + + + + + + + + + + 268435456 + true + + diff --git a/deployments/signoz/clickhouse/custom-function.xml b/deployments/signoz/clickhouse/custom-function.xml new file mode 100644 index 00000000..b2b3f91a --- /dev/null +++ b/deployments/signoz/clickhouse/custom-function.xml @@ -0,0 +1,21 @@ + + + executable + histogramQuantile + Float64 + + Array(Float64) + buckets + + + Array(Float64) + counts + + + Float64 + quantile + + CSV + ./histogramQuantile + + diff --git a/deployments/signoz/clickhouse/user_scripts/.gitkeep b/deployments/signoz/clickhouse/user_scripts/.gitkeep new file mode 100644 index 00000000..e69de29b diff --git a/deployments/signoz/clickhouse/users.xml b/deployments/signoz/clickhouse/users.xml new file mode 100644 index 00000000..f1856207 --- /dev/null +++ b/deployments/signoz/clickhouse/users.xml @@ -0,0 +1,123 @@ + + + + + + + + + + 10000000000 + + + random + + + + + 1 + + + + + + + + + + + + + ::/0 + + + + default + + + default + + + + + + + + + + + + + + 3600 + + + 0 + 0 + 0 + 0 + 0 + + + + diff --git a/deployments/signoz/docker-compose.yaml b/deployments/signoz/docker-compose.yaml index 4ca2b4a7..b69f3cb3 100644 --- a/deployments/signoz/docker-compose.yaml +++ b/deployments/signoz/docker-compose.yaml @@ -1,84 +1,182 @@ +# Self-contained SigNoz stack for running an OTel-native observability backend +# locally. Modelled on the upstream SigNoz `deploy/docker/docker-compose.yaml` +# (SigNoz v0.122.0 / signoz-otel-collector v0.144.3), trimmed to a single node +# and adapted to this repo's layout. +# +# Bring it up: +# docker compose -f deployments/signoz/docker-compose.yaml up -d +# +# UI: http://localhost:3301 (host 3301 → container 8080; WaveHouse owns 8080) +# OTLP gRPC: localhost:4317 (point WaveHouse at it: WH_OTEL_ADDR=127.0.0.1:4317, +# or host.docker.internal:4317 from another container) +# OTLP HTTP: localhost:4318 +# +# Long-running services: clickhouse, zookeeper-1, signoz, otel-collector. +# One-shot init containers (normal to see them as `Exited (0)`): +# - init-clickhouse: fetches the histogramQuantile UDF binary ClickHouse loads +# - signoz-telemetrystore-migrator: creates the signoz_* ClickHouse schema, then exits + +x-common: &common + networks: + - signoz-net + restart: unless-stopped + logging: + options: + max-size: 50m + max-file: "3" + +x-clickhouse-defaults: &clickhouse-defaults + <<: *common + image: clickhouse/clickhouse-server:25.5.6 + tty: true + depends_on: + init-clickhouse: + condition: service_completed_successfully + zookeeper-1: + condition: service_healthy + healthcheck: + test: ["CMD", "wget", "--spider", "-q", "0.0.0.0:8123/ping"] + interval: 30s + timeout: 5s + retries: 3 + ulimits: + nproc: 65535 + nofile: + soft: 262144 + hard: 262144 + environment: + - CLICKHOUSE_SKIP_USER_SETUP=1 + +x-db-depend: &db-depend + <<: *common + depends_on: + clickhouse: + condition: service_healthy + services: - volume-fixer: - image: alpine - container_name: signoz-volume-fixer - user: root + init-clickhouse: + <<: *common + image: clickhouse/clickhouse-server:25.5.6 + container_name: signoz-init-clickhouse + restart: on-failure + command: + - bash + - -c + - | + version="v0.0.1" + node_os=$$(uname -s | tr '[:upper:]' '[:lower:]') + node_arch=$$(uname -m | sed s/aarch64/arm64/ | sed s/x86_64/amd64/) + echo "Fetching histogram-quantile binary for $${node_os}/$${node_arch}" + cd /tmp + wget -O histogram-quantile.tar.gz "https://github.com/SigNoz/signoz/releases/download/histogram-quantile%2F$${version}/histogram-quantile_$${node_os}_$${node_arch}.tar.gz" + tar -xvzf histogram-quantile.tar.gz + mv histogram-quantile /var/lib/clickhouse/user_scripts/histogramQuantile volumes: - - clickhouse_data:/mnt/ch_data - - signoz_db:/mnt/sz_db - command: > - sh -c "chown -R 101:101 /mnt/ch_data && - chown -R 1000:1000 /mnt/sz_db && - echo 'Permissions fixed'" + - clickhouse-scripts:/var/lib/clickhouse/user_scripts/ - clickhouse: - image: clickhouse/clickhouse-server:24.1.2-alpine - container_name: signoz-clickhouse - depends_on: - volume-fixer: - condition: service_completed_successfully + zookeeper-1: + <<: *common + image: signoz/zookeeper:3.7.1 + container_name: signoz-zookeeper-1 + user: root environment: - - CLICKHOUSE_USER=${SIGNOZ_CH_USER:-default} - - CLICKHOUSE_PASSWORD=${SIGNOZ_CH_PASSWORD:-password} + - ZOO_SERVER_ID=1 + - ALLOW_ANONYMOUS_LOGIN=yes + - ZOO_AUTOPURGE_INTERVAL=1 + - ZOO_ENABLE_PROMETHEUS_METRICS=yes + - ZOO_PROMETHEUS_METRICS_PORT_NUMBER=9141 volumes: - - clickhouse_data:/var/lib/clickhouse + - zookeeper-1:/bitnami/zookeeper healthcheck: - test: ["CMD", "wget", "--spider", "-q", "http://127.0.0.1:8123/ping"] - interval: 10s + test: ["CMD-SHELL", "curl -s -m 2 http://localhost:8080/commands/ruok | grep error | grep null"] + interval: 30s timeout: 5s - retries: 10 - start_period: 20s + retries: 3 - migrator: - image: signoz/signoz-otel-collector:latest - container_name: signoz-migrator - depends_on: - clickhouse: - condition: service_healthy - environment: - - SIGNOZ_OTEL_COLLECTOR_CLICKHOUSE_DSN=tcp://${SIGNOZ_CH_USER:-default}:${SIGNOZ_CH_PASSWORD:-password}@clickhouse:9000 - entrypoint: ["/bin/sh"] - command: - - -c - - | - /signoz-otel-collector migrate bootstrap --clickhouse-cluster="" --clickhouse-replication=false && - /signoz-otel-collector migrate sync up --clickhouse-cluster="" --clickhouse-replication=false && - /signoz-otel-collector migrate async up --clickhouse-cluster="" --clickhouse-replication=false + clickhouse: + <<: *clickhouse-defaults + container_name: signoz-clickhouse + # Ports are intentionally not published — this ClickHouse is internal to + # SigNoz. (WaveHouse has its own ClickHouse in deployments/compose/.) + volumes: + - ./clickhouse/config.xml:/etc/clickhouse-server/config.xml + - ./clickhouse/users.xml:/etc/clickhouse-server/users.xml + - ./clickhouse/custom-function.xml:/etc/clickhouse-server/custom-function.xml + - ./clickhouse/cluster.xml:/etc/clickhouse-server/config.d/cluster.xml + - clickhouse-scripts:/var/lib/clickhouse/user_scripts/ + - clickhouse:/var/lib/clickhouse/ signoz: - image: signoz/query-service:latest + <<: *db-depend + image: signoz/signoz:v0.122.0 container_name: signoz - depends_on: - migrator: - condition: service_completed_successfully ports: - - "3301:8080" - environment: - - SIGNOZ_PORT=8080 - - FRONTEND_API_ENDPOINT=http://localhost:3301 - - SIGNOZ_TELEMETRYSTORE_CLICKHOUSE_DSN=tcp://${SIGNOZ_CH_USER:-default}:${SIGNOZ_CH_PASSWORD:-password}@clickhouse:9000 + - "3301:8080" # UI + API (8080 in the container; published on 3301 — WaveHouse owns host 8080) volumes: - - signoz_db:/var/lib/signoz + - sqlite:/var/lib/signoz/ + environment: + - SIGNOZ_ALERTMANAGER_PROVIDER=signoz + - SIGNOZ_TELEMETRYSTORE_CLICKHOUSE_DSN=tcp://clickhouse:9000 + - SIGNOZ_SQLSTORE_SQLITE_PATH=/var/lib/signoz/signoz.db + - SIGNOZ_TOKENIZER_JWT_SECRET=secret + healthcheck: + test: ["CMD", "wget", "--spider", "-q", "localhost:8080/api/v1/health"] + interval: 30s + timeout: 5s + retries: 3 otel-collector: - image: signoz/signoz-otel-collector:latest + <<: *db-depend + image: signoz/signoz-otel-collector:v0.144.3 container_name: signoz-otel-collector - depends_on: - migrator: - condition: service_completed_successfully - environment: - - SIGNOZ_CH_USER=${SIGNOZ_CH_USER:-default} - - SIGNOZ_CH_PASSWORD=${SIGNOZ_CH_PASSWORD:-password} + entrypoint: ["/bin/sh"] + command: + - -c + - | + /signoz-otel-collector migrate sync check && + /signoz-otel-collector --config=/etc/otel-collector-config.yaml --manager-config=/etc/manager-config.yaml --copy-path=/var/tmp/collector-config.yaml volumes: - ./otel-collector-config.yaml:/etc/otel-collector-config.yaml + - ./otel-collector-opamp-config.yaml:/etc/manager-config.yaml + environment: + - OTEL_RESOURCE_ATTRIBUTES=host.name=signoz-host,os.type=linux + - LOW_CARDINAL_EXCEPTION_GROUPING=false + - SIGNOZ_OTEL_COLLECTOR_CLICKHOUSE_DSN=tcp://clickhouse:9000 + - SIGNOZ_OTEL_COLLECTOR_CLICKHOUSE_CLUSTER=cluster + - SIGNOZ_OTEL_COLLECTOR_CLICKHOUSE_REPLICATION=true + - SIGNOZ_OTEL_COLLECTOR_TIMEOUT=10m ports: - - "4317:4317" - command: ["--config=/etc/otel-collector-config.yaml"] + - "4317:4317" # OTLP gRPC receiver — WaveHouse pushes here + - "4318:4318" # OTLP HTTP receiver -volumes: - clickhouse_data: - signoz_db: + signoz-telemetrystore-migrator: + <<: *db-depend + image: signoz/signoz-otel-collector:v0.144.3 + container_name: signoz-telemetrystore-migrator + restart: on-failure + entrypoint: ["/bin/sh"] + command: + - -c + - | + /signoz-otel-collector migrate bootstrap && + /signoz-otel-collector migrate sync up && + /signoz-otel-collector migrate async up + environment: + - SIGNOZ_OTEL_COLLECTOR_CLICKHOUSE_DSN=tcp://clickhouse:9000 + - SIGNOZ_OTEL_COLLECTOR_CLICKHOUSE_CLUSTER=cluster + - SIGNOZ_OTEL_COLLECTOR_CLICKHOUSE_REPLICATION=true + - SIGNOZ_OTEL_COLLECTOR_TIMEOUT=10m networks: - default: - name: wavehouse-obs-net \ No newline at end of file + signoz-net: + name: signoz-net + +volumes: + clickhouse: + name: signoz-clickhouse + clickhouse-scripts: + name: signoz-clickhouse-user-scripts + sqlite: + name: signoz-sqlite + zookeeper-1: + name: signoz-zookeeper-1 diff --git a/deployments/signoz/otel-collector-config.yaml b/deployments/signoz/otel-collector-config.yaml index c88f640e..ee7724c9 100644 --- a/deployments/signoz/otel-collector-config.yaml +++ b/deployments/signoz/otel-collector-config.yaml @@ -68,24 +68,24 @@ extensions: endpoint: 0.0.0.0:1777 exporters: clickhousetraces: - datasource: tcp://${env:SIGNOZ_CH_USER}:${env:SIGNOZ_CH_PASSWORD}@clickhouse:9000/signoz_traces + datasource: tcp://clickhouse:9000/signoz_traces low_cardinal_exception_grouping: ${env:LOW_CARDINAL_EXCEPTION_GROUPING} use_new_schema: true signozclickhousemetrics: - dsn: tcp://${env:SIGNOZ_CH_USER}:${env:SIGNOZ_CH_PASSWORD}@clickhouse:9000/signoz_metrics + dsn: tcp://clickhouse:9000/signoz_metrics clickhouselogsexporter: - dsn: tcp://${env:SIGNOZ_CH_USER}:${env:SIGNOZ_CH_PASSWORD}@clickhouse:9000/signoz_logs + dsn: tcp://clickhouse:9000/signoz_logs timeout: 10s use_new_schema: true signozclickhousemeter: - dsn: tcp://${env:SIGNOZ_CH_USER}:${env:SIGNOZ_CH_PASSWORD}@clickhouse:9000/signoz_meter + dsn: tcp://clickhouse:9000/signoz_meter timeout: 45s sending_queue: enabled: false metadataexporter: cache: provider: in_memory - dsn: tcp://${env:SIGNOZ_CH_USER}:${env:SIGNOZ_CH_PASSWORD}@clickhouse:9000/signoz_metadata + dsn: tcp://clickhouse:9000/signoz_metadata enabled: true timeout: 45s service: diff --git a/deployments/signoz/otel-collector-opamp-config.yaml b/deployments/signoz/otel-collector-opamp-config.yaml new file mode 100644 index 00000000..72676077 --- /dev/null +++ b/deployments/signoz/otel-collector-opamp-config.yaml @@ -0,0 +1 @@ +server_endpoint: ws://signoz:4320/v1/opamp diff --git a/docs/src/content/docs/deployment.md b/docs/src/content/docs/deployment.md index 6f5a2ac8..da72d56e 100644 --- a/docs/src/content/docs/deployment.md +++ b/docs/src/content/docs/deployment.md @@ -373,15 +373,13 @@ By default, `prometheus.port` is `0`, which mounts `/metrics` on the main API se For production posture where metrics should not be exposed on the public API listener, set `port` to a separate non-zero value (e.g. `9091`). WaveHouse spins up a dedicated HTTP listener bound to that port serving only `/metrics`. Firewall the port to internal networks only; the main API listener stays where it was. Both listeners participate in graceful shutdown. -`deployments/signoz/` is a self-contained Docker Compose setup for running SigNoz locally (ClickHouse + query service + OTel collector at `:4317`). Bring it up: +`deployments/signoz/` is a self-contained Docker Compose setup for running SigNoz locally. It is modelled on the upstream SigNoz `deploy/docker/docker-compose.yaml` (SigNoz `v0.122.0` / `signoz-otel-collector v0.144.3`), trimmed to a single node: ClickHouse + ZooKeeper + the SigNoz query/UI service + the OTel collector (OTLP gRPC on `:4317`, HTTP on `:4318`), plus two one-shot init containers (`signoz-init-clickhouse` fetches the `histogramQuantile` UDF binary; `signoz-telemetrystore-migrator` creates the `signoz_*` ClickHouse schema). Both init containers exit `0` after running — that's expected. Bring it up: ```bash docker compose -f deployments/signoz/docker-compose.yaml up -d ``` -ClickHouse credentials inside the SigNoz stack default to `default` / `password`. To override, copy `deployments/signoz/.env.example` to `deployments/signoz/.env` and set `SIGNOZ_CH_USER` / `SIGNOZ_CH_PASSWORD`. The `.env` file is gitignored. - -The SigNoz UI is exposed on `http://localhost:3301`. Point WaveHouse at the collector with `WH_OTEL_ADDR=127.0.0.1:4317` (the default). +The SigNoz UI is on `http://localhost:3301` (host `3301` → container `8080`; `8080` is left for WaveHouse). Point WaveHouse at the collector with `WH_OTEL_ADDR=127.0.0.1:4317` for a host-side WaveHouse (the default), or `WH_OTEL_ADDR=host.docker.internal:4317` from a WaveHouse container — the collector publishes `4317` on the host. See `deployments/compose/standalone.signoz.yaml` for a compose override that wires the containerized `standalone.yaml` WaveHouse into this stack. ## Resetting Data in Development From bf59b8c21b400b03554bf899d2f19c8855a4c5e2 Mon Sep 17 00:00:00 2001 From: Jack Woods Date: Tue, 12 May 2026 23:57:41 -0400 Subject: [PATCH 03/25] chore(deploy): version-controlled SigNoz dashboards + loader MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add two WaveHouse dashboards as JSON under deployments/signoz/dashboards/ (SigNoz query-builder v5 schema): - wavehouse-overview.json — HTTP request rate / status / route, latency p50–p99, span call rate by operation, OTLP intake at the collector - wavehouse-runtime-internals.json — Go runtime, embedded NATS, ingest pipeline (events/s, DLQ drops), HTTP payload sizes Plus deployments/signoz/load-dashboards.sh: upserts the JSON files into a running SigNoz by title (PUT existing, POST new), authenticating via SIGNOZ_EMAIL/SIGNOZ_PASSWORD or a SIGNOZ_TOKEN. SigNoz OSS has no on-disk dashboard provisioning and disables self-registration, so loading is a one-time-per-person step after creating the account in the UI — documented in docs/deployment.md. Local dev tooling only — on the local-deploy-stack branch, not the observability PR. Co-Authored-By: Claude Opus 4.7 (1M context) --- deployments/signoz/dashboards/README.md | 31 + .../signoz/dashboards/wavehouse-overview.json | 523 ++++++++++++++++ .../wavehouse-runtime-internals.json | 576 ++++++++++++++++++ deployments/signoz/load-dashboards.sh | 76 +++ docs/src/content/docs/deployment.md | 16 + 5 files changed, 1222 insertions(+) create mode 100644 deployments/signoz/dashboards/README.md create mode 100644 deployments/signoz/dashboards/wavehouse-overview.json create mode 100644 deployments/signoz/dashboards/wavehouse-runtime-internals.json create mode 100755 deployments/signoz/load-dashboards.sh diff --git a/deployments/signoz/dashboards/README.md b/deployments/signoz/dashboards/README.md new file mode 100644 index 00000000..a0699858 --- /dev/null +++ b/deployments/signoz/dashboards/README.md @@ -0,0 +1,31 @@ +# SigNoz dashboards (WaveHouse) + +Version-controlled SigNoz dashboard definitions for the local observability stack. + +| File | What it shows | +|---|---| +| `wavehouse-overview.json` | HTTP request rate / status / route, request latency p50–p99, span call rate by operation, OTLP intake at the collector | +| `wavehouse-runtime-internals.json` | Go runtime (goroutines, memory, allocations), embedded NATS (connections, msg/s), ingest pipeline (events/s, DLQ drops), HTTP payload sizes | + +Format: SigNoz query-builder **v5** schema (the `version` field). Each file is exactly the `.data.data` object the SigNoz dashboards API returns. + +## Loading them + +SigNoz OSS doesn't provision dashboards from disk, so after the stack is up and you've created your account in the UI: + +```bash +SIGNOZ_EMAIL=you@example.com SIGNOZ_PASSWORD='…' ../load-dashboards.sh +``` + +`../load-dashboards.sh` upserts by title (existing dashboards are updated in place), so re-run it whenever these files change. See its header for auth options (`SIGNOZ_TOKEN`, `SIGNOZ_URL`). + +## Editing a dashboard + +Easiest path: edit it in the SigNoz UI, then export it back — + +```bash +curl -s -H "Authorization: Bearer $SIGNOZ_TOKEN" \ + http://localhost:3301/api/v1/dashboards/ | jq '.data.data' > wavehouse-overview.json +``` + +— and commit the result. diff --git a/deployments/signoz/dashboards/wavehouse-overview.json b/deployments/signoz/dashboards/wavehouse-overview.json new file mode 100644 index 00000000..c02f573f --- /dev/null +++ b/deployments/signoz/dashboards/wavehouse-overview.json @@ -0,0 +1,523 @@ +{ + "title": "WaveHouse · Overview", + "description": "HTTP traffic, latency, and OTLP intake for the local WaveHouse service.", + "tags": [ + "wavehouse" + ], + "variables": {}, + "version": "v5", + "layout": [ + { + "i": "fcfd52ba-4ada-55ee-b1d9-6edca38a086a", + "x": 0, + "y": 0, + "w": 6, + "h": 3 + }, + { + "i": "840b8f2b-73b6-5156-ac06-329687f0a3b3", + "x": 6, + "y": 0, + "w": 6, + "h": 3 + }, + { + "i": "611d6a95-e20b-5129-b415-b360a734d1d0", + "x": 0, + "y": 3, + "w": 6, + "h": 3 + }, + { + "i": "30eb2113-5c9b-5f79-b40e-5100a4eae76d", + "x": 6, + "y": 3, + "w": 6, + "h": 3 + }, + { + "i": "bcd0ddb5-20de-5f3d-b279-c9f12399a4f0", + "x": 0, + "y": 6, + "w": 6, + "h": 3 + }, + { + "i": "cd54ba63-4ae5-5825-916a-d4978e73e6aa", + "x": 0, + "y": 9, + "w": 12, + "h": 3 + } + ], + "widgets": [ + { + "id": "fcfd52ba-4ada-55ee-b1d9-6edca38a086a", + "title": "HTTP request rate (req/s)", + "description": "rate(http.server.request.duration.count) — all WaveHouse HTTP requests", + "panelTypes": "graph", + "timePreferance": "GLOBAL_TIME", + "query": { + "queryType": "builder", + "builder": { + "queryData": [ + { + "aggregations": [ + { + "metricName": "http.server.request.duration.count", + "reduceTo": "avg", + "spaceAggregation": "sum", + "temporality": null, + "timeAggregation": "rate" + } + ], + "dataSource": "metrics", + "disabled": false, + "expression": "A", + "filter": { + "expression": "" + }, + "groupBy": [], + "having": { + "expression": "" + }, + "legend": "req/s", + "orderBy": [], + "queryName": "A", + "stepInterval": 60 + } + ], + "queryFormulas": [] + }, + "clickhouse_sql": [ + { + "name": "A", + "query": "", + "legend": "", + "disabled": false + } + ], + "promql": [ + { + "name": "A", + "query": "", + "legend": "", + "disabled": false + } + ], + "id": "861fb838-5028-5fb4-8738-bb60efefa176" + }, + "yAxisUnit": "reqps" + }, + { + "id": "840b8f2b-73b6-5156-ac06-329687f0a3b3", + "title": "HTTP request rate by status code", + "description": "", + "panelTypes": "graph", + "timePreferance": "GLOBAL_TIME", + "query": { + "queryType": "builder", + "builder": { + "queryData": [ + { + "aggregations": [ + { + "metricName": "http.server.request.duration.count", + "reduceTo": "avg", + "spaceAggregation": "sum", + "temporality": null, + "timeAggregation": "rate" + } + ], + "dataSource": "metrics", + "disabled": false, + "expression": "A", + "filter": { + "expression": "" + }, + "groupBy": [ + { + "key": "http.response.status_code", + "dataType": "string", + "type": "tag", + "isColumn": false + } + ], + "having": { + "expression": "" + }, + "legend": "{{http.response.status_code}}", + "orderBy": [], + "queryName": "A", + "stepInterval": 60 + } + ], + "queryFormulas": [] + }, + "clickhouse_sql": [ + { + "name": "A", + "query": "", + "legend": "", + "disabled": false + } + ], + "promql": [ + { + "name": "A", + "query": "", + "legend": "", + "disabled": false + } + ], + "id": "f052edde-5eb9-58d9-8c35-e04f530c3f1e" + }, + "yAxisUnit": "reqps" + }, + { + "id": "611d6a95-e20b-5129-b415-b360a734d1d0", + "title": "HTTP latency p50 / p95 / p99 (s)", + "description": "", + "panelTypes": "graph", + "timePreferance": "GLOBAL_TIME", + "query": { + "queryType": "builder", + "builder": { + "queryData": [ + { + "aggregations": [ + { + "metricName": "http.server.request.duration.bucket", + "reduceTo": "avg", + "spaceAggregation": "p50", + "temporality": null, + "timeAggregation": "rate" + } + ], + "dataSource": "metrics", + "disabled": false, + "expression": "A", + "filter": { + "expression": "" + }, + "groupBy": [], + "having": { + "expression": "" + }, + "legend": "p50", + "orderBy": [], + "queryName": "A", + "stepInterval": 60 + }, + { + "aggregations": [ + { + "metricName": "http.server.request.duration.bucket", + "reduceTo": "avg", + "spaceAggregation": "p95", + "temporality": null, + "timeAggregation": "rate" + } + ], + "dataSource": "metrics", + "disabled": false, + "expression": "B", + "filter": { + "expression": "" + }, + "groupBy": [], + "having": { + "expression": "" + }, + "legend": "p95", + "orderBy": [], + "queryName": "B", + "stepInterval": 60 + }, + { + "aggregations": [ + { + "metricName": "http.server.request.duration.bucket", + "reduceTo": "avg", + "spaceAggregation": "p99", + "temporality": null, + "timeAggregation": "rate" + } + ], + "dataSource": "metrics", + "disabled": false, + "expression": "C", + "filter": { + "expression": "" + }, + "groupBy": [], + "having": { + "expression": "" + }, + "legend": "p99", + "orderBy": [], + "queryName": "C", + "stepInterval": 60 + } + ], + "queryFormulas": [] + }, + "clickhouse_sql": [ + { + "name": "A", + "query": "", + "legend": "", + "disabled": false + } + ], + "promql": [ + { + "name": "A", + "query": "", + "legend": "", + "disabled": false + } + ], + "id": "f26868b0-8b6f-5b52-85d3-3ef8ccc500e3" + }, + "yAxisUnit": "s" + }, + { + "id": "30eb2113-5c9b-5f79-b40e-5100a4eae76d", + "title": "HTTP request rate by route", + "description": "", + "panelTypes": "graph", + "timePreferance": "GLOBAL_TIME", + "query": { + "queryType": "builder", + "builder": { + "queryData": [ + { + "aggregations": [ + { + "metricName": "http.server.request.duration.count", + "reduceTo": "avg", + "spaceAggregation": "sum", + "temporality": null, + "timeAggregation": "rate" + } + ], + "dataSource": "metrics", + "disabled": false, + "expression": "A", + "filter": { + "expression": "" + }, + "groupBy": [ + { + "key": "http.route", + "dataType": "string", + "type": "tag", + "isColumn": false + } + ], + "having": { + "expression": "" + }, + "legend": "{{http.route}}", + "orderBy": [], + "queryName": "A", + "stepInterval": 60 + } + ], + "queryFormulas": [] + }, + "clickhouse_sql": [ + { + "name": "A", + "query": "", + "legend": "", + "disabled": false + } + ], + "promql": [ + { + "name": "A", + "query": "", + "legend": "", + "disabled": false + } + ], + "id": "8998dfbc-f2bb-5d09-8f47-6c846e5ba4a2" + }, + "yAxisUnit": "reqps" + }, + { + "id": "bcd0ddb5-20de-5f3d-b279-c9f12399a4f0", + "title": "Span call rate by operation (signoz_calls_total)", + "description": "All spans, including internal (clickhouse_insert, schema refresh, NATS, …)", + "panelTypes": "graph", + "timePreferance": "GLOBAL_TIME", + "query": { + "queryType": "builder", + "builder": { + "queryData": [ + { + "aggregations": [ + { + "metricName": "signoz_calls_total", + "reduceTo": "avg", + "spaceAggregation": "sum", + "temporality": null, + "timeAggregation": "rate" + } + ], + "dataSource": "metrics", + "disabled": false, + "expression": "A", + "filter": { + "expression": "" + }, + "groupBy": [ + { + "key": "operation", + "dataType": "string", + "type": "tag", + "isColumn": false + } + ], + "having": { + "expression": "" + }, + "legend": "{{operation}}", + "orderBy": [], + "queryName": "A", + "stepInterval": 60 + } + ], + "queryFormulas": [] + }, + "clickhouse_sql": [ + { + "name": "A", + "query": "", + "legend": "", + "disabled": false + } + ], + "promql": [ + { + "name": "A", + "query": "", + "legend": "", + "disabled": false + } + ], + "id": "be4a15ec-7da7-5d92-b4d0-dc24be93d226" + }, + "yAxisUnit": "reqps" + }, + { + "id": "cd54ba63-4ae5-5825-916a-d4978e73e6aa", + "title": "Spans / metrics / logs received by the collector (per sec)", + "description": "OTLP intake at the SigNoz collector — a quick \"is data flowing\" check", + "panelTypes": "graph", + "timePreferance": "GLOBAL_TIME", + "query": { + "queryType": "builder", + "builder": { + "queryData": [ + { + "aggregations": [ + { + "metricName": "otelcol_receiver_accepted_spans", + "reduceTo": "avg", + "spaceAggregation": "sum", + "temporality": null, + "timeAggregation": "rate" + } + ], + "dataSource": "metrics", + "disabled": false, + "expression": "A", + "filter": { + "expression": "" + }, + "groupBy": [], + "having": { + "expression": "" + }, + "legend": "spans/s", + "orderBy": [], + "queryName": "A", + "stepInterval": 60 + }, + { + "aggregations": [ + { + "metricName": "otelcol_receiver_accepted_metric_points", + "reduceTo": "avg", + "spaceAggregation": "sum", + "temporality": null, + "timeAggregation": "rate" + } + ], + "dataSource": "metrics", + "disabled": false, + "expression": "B", + "filter": { + "expression": "" + }, + "groupBy": [], + "having": { + "expression": "" + }, + "legend": "metric points/s", + "orderBy": [], + "queryName": "B", + "stepInterval": 60 + }, + { + "aggregations": [ + { + "metricName": "otelcol_receiver_accepted_log_records", + "reduceTo": "avg", + "spaceAggregation": "sum", + "temporality": null, + "timeAggregation": "rate" + } + ], + "dataSource": "metrics", + "disabled": false, + "expression": "C", + "filter": { + "expression": "" + }, + "groupBy": [], + "having": { + "expression": "" + }, + "legend": "log records/s", + "orderBy": [], + "queryName": "C", + "stepInterval": 60 + } + ], + "queryFormulas": [] + }, + "clickhouse_sql": [ + { + "name": "A", + "query": "", + "legend": "", + "disabled": false + } + ], + "promql": [ + { + "name": "A", + "query": "", + "legend": "", + "disabled": false + } + ], + "id": "3350c7df-3415-56b0-a39f-f263405dfb98" + } + } + ] +} diff --git a/deployments/signoz/dashboards/wavehouse-runtime-internals.json b/deployments/signoz/dashboards/wavehouse-runtime-internals.json new file mode 100644 index 00000000..48dc5259 --- /dev/null +++ b/deployments/signoz/dashboards/wavehouse-runtime-internals.json @@ -0,0 +1,576 @@ +{ + "title": "WaveHouse · Runtime & Internals", + "description": "Go runtime, embedded NATS, ingest pipeline, and HTTP payload sizes for the local WaveHouse service.", + "tags": [ + "wavehouse" + ], + "variables": {}, + "version": "v5", + "layout": [ + { + "i": "cd634115-f28a-5eac-9139-47f084d17325", + "x": 0, + "y": 0, + "w": 4, + "h": 3 + }, + { + "i": "e9fa7d35-09e8-54bc-a95f-6f6d43bf0e34", + "x": 4, + "y": 0, + "w": 4, + "h": 3 + }, + { + "i": "596b38da-a5d3-5aae-96de-054fd50cfbf9", + "x": 8, + "y": 0, + "w": 4, + "h": 3 + }, + { + "i": "05167217-93f9-5475-acdc-3317beb3c1c7", + "x": 0, + "y": 3, + "w": 4, + "h": 3 + }, + { + "i": "5d29c83c-0940-5d76-a498-f5af98e7af07", + "x": 4, + "y": 3, + "w": 4, + "h": 3 + }, + { + "i": "f7c79930-5bdd-5fb1-a914-6433a2cf0743", + "x": 8, + "y": 3, + "w": 4, + "h": 3 + }, + { + "i": "4c82c49e-c310-5bec-8065-49689e17e8c9", + "x": 0, + "y": 6, + "w": 6, + "h": 3 + }, + { + "i": "8828b51f-f0fc-56b2-a674-21d95d230501", + "x": 6, + "y": 6, + "w": 6, + "h": 3 + } + ], + "widgets": [ + { + "id": "cd634115-f28a-5eac-9139-47f084d17325", + "title": "Go — goroutines", + "description": "", + "panelTypes": "graph", + "timePreferance": "GLOBAL_TIME", + "query": { + "queryType": "builder", + "builder": { + "queryData": [ + { + "aggregations": [ + { + "metricName": "go.goroutine.count", + "reduceTo": "avg", + "spaceAggregation": "sum", + "temporality": null, + "timeAggregation": "latest" + } + ], + "dataSource": "metrics", + "disabled": false, + "expression": "A", + "filter": { + "expression": "" + }, + "groupBy": [], + "having": { + "expression": "" + }, + "legend": "goroutines", + "orderBy": [], + "queryName": "A", + "stepInterval": 60 + } + ], + "queryFormulas": [] + }, + "clickhouse_sql": [ + { + "name": "A", + "query": "", + "legend": "", + "disabled": false + } + ], + "promql": [ + { + "name": "A", + "query": "", + "legend": "", + "disabled": false + } + ], + "id": "a85cb74f-054f-54c2-abeb-dfc9c6818279" + }, + "yAxisUnit": "short" + }, + { + "id": "e9fa7d35-09e8-54bc-a95f-6f6d43bf0e34", + "title": "Go — memory used (bytes)", + "description": "", + "panelTypes": "graph", + "timePreferance": "GLOBAL_TIME", + "query": { + "queryType": "builder", + "builder": { + "queryData": [ + { + "aggregations": [ + { + "metricName": "go.memory.used", + "reduceTo": "avg", + "spaceAggregation": "sum", + "temporality": null, + "timeAggregation": "latest" + } + ], + "dataSource": "metrics", + "disabled": false, + "expression": "A", + "filter": { + "expression": "" + }, + "groupBy": [ + { + "key": "go.memory.type", + "dataType": "string", + "type": "tag", + "isColumn": false + } + ], + "having": { + "expression": "" + }, + "legend": "{{go.memory.type}}", + "orderBy": [], + "queryName": "A", + "stepInterval": 60 + } + ], + "queryFormulas": [] + }, + "clickhouse_sql": [ + { + "name": "A", + "query": "", + "legend": "", + "disabled": false + } + ], + "promql": [ + { + "name": "A", + "query": "", + "legend": "", + "disabled": false + } + ], + "id": "2678fb88-02cb-50b5-9456-a15e6229f024" + }, + "yAxisUnit": "bytes" + }, + { + "id": "596b38da-a5d3-5aae-96de-054fd50cfbf9", + "title": "Go — allocation rate (bytes/s)", + "description": "", + "panelTypes": "graph", + "timePreferance": "GLOBAL_TIME", + "query": { + "queryType": "builder", + "builder": { + "queryData": [ + { + "aggregations": [ + { + "metricName": "go.memory.allocations", + "reduceTo": "avg", + "spaceAggregation": "sum", + "temporality": null, + "timeAggregation": "rate" + } + ], + "dataSource": "metrics", + "disabled": false, + "expression": "A", + "filter": { + "expression": "" + }, + "groupBy": [], + "having": { + "expression": "" + }, + "legend": "bytes/s", + "orderBy": [], + "queryName": "A", + "stepInterval": 60 + } + ], + "queryFormulas": [] + }, + "clickhouse_sql": [ + { + "name": "A", + "query": "", + "legend": "", + "disabled": false + } + ], + "promql": [ + { + "name": "A", + "query": "", + "legend": "", + "disabled": false + } + ], + "id": "c39d7174-e9d2-5b0b-b43e-cfb2b575d403" + }, + "yAxisUnit": "Bps" + }, + { + "id": "05167217-93f9-5475-acdc-3317beb3c1c7", + "title": "NATS — client connections", + "description": "", + "panelTypes": "graph", + "timePreferance": "GLOBAL_TIME", + "query": { + "queryType": "builder", + "builder": { + "queryData": [ + { + "aggregations": [ + { + "metricName": "wavehouse_nats_connections", + "reduceTo": "avg", + "spaceAggregation": "sum", + "temporality": null, + "timeAggregation": "latest" + } + ], + "dataSource": "metrics", + "disabled": false, + "expression": "A", + "filter": { + "expression": "" + }, + "groupBy": [], + "having": { + "expression": "" + }, + "legend": "connections", + "orderBy": [], + "queryName": "A", + "stepInterval": 60 + } + ], + "queryFormulas": [] + }, + "clickhouse_sql": [ + { + "name": "A", + "query": "", + "legend": "", + "disabled": false + } + ], + "promql": [ + { + "name": "A", + "query": "", + "legend": "", + "disabled": false + } + ], + "id": "7d74c1ba-a8a9-5dad-bcbb-0b7c76d5fcdd" + }, + "yAxisUnit": "short" + }, + { + "id": "5d29c83c-0940-5d76-a498-f5af98e7af07", + "title": "NATS — messages in (msg/s)", + "description": "", + "panelTypes": "graph", + "timePreferance": "GLOBAL_TIME", + "query": { + "queryType": "builder", + "builder": { + "queryData": [ + { + "aggregations": [ + { + "metricName": "wavehouse_nats_in_msgs_total", + "reduceTo": "avg", + "spaceAggregation": "sum", + "temporality": null, + "timeAggregation": "rate" + } + ], + "dataSource": "metrics", + "disabled": false, + "expression": "A", + "filter": { + "expression": "" + }, + "groupBy": [], + "having": { + "expression": "" + }, + "legend": "msg/s", + "orderBy": [], + "queryName": "A", + "stepInterval": 60 + } + ], + "queryFormulas": [] + }, + "clickhouse_sql": [ + { + "name": "A", + "query": "", + "legend": "", + "disabled": false + } + ], + "promql": [ + { + "name": "A", + "query": "", + "legend": "", + "disabled": false + } + ], + "id": "42fd6f7b-2f45-5b2a-bd27-29e1fa6818a4" + } + }, + { + "id": "f7c79930-5bdd-5fb1-a914-6433a2cf0743", + "title": "Ingest — events written to ClickHouse (events/s)", + "description": "Populates once ingest traffic flows (WaveHouse uses bring-your-own-schema).", + "panelTypes": "graph", + "timePreferance": "GLOBAL_TIME", + "query": { + "queryType": "builder", + "builder": { + "queryData": [ + { + "aggregations": [ + { + "metricName": "wavehouse_bento_events_processed", + "reduceTo": "avg", + "spaceAggregation": "sum", + "temporality": null, + "timeAggregation": "rate" + } + ], + "dataSource": "metrics", + "disabled": false, + "expression": "A", + "filter": { + "expression": "" + }, + "groupBy": [ + { + "key": "table", + "dataType": "string", + "type": "tag", + "isColumn": false + } + ], + "having": { + "expression": "" + }, + "legend": "{{table}}", + "orderBy": [], + "queryName": "A", + "stepInterval": 60 + } + ], + "queryFormulas": [] + }, + "clickhouse_sql": [ + { + "name": "A", + "query": "", + "legend": "", + "disabled": false + } + ], + "promql": [ + { + "name": "A", + "query": "", + "legend": "", + "disabled": false + } + ], + "id": "b6d9577d-10ac-55be-97ba-7e6d4afc5ffa" + } + }, + { + "id": "4c82c49e-c310-5bec-8065-49689e17e8c9", + "title": "Ingest — DLQ messages permanently dropped (per sec)", + "description": "", + "panelTypes": "graph", + "timePreferance": "GLOBAL_TIME", + "query": { + "queryType": "builder", + "builder": { + "queryData": [ + { + "aggregations": [ + { + "metricName": "wavehouse_bento_dlq_dropped", + "reduceTo": "avg", + "spaceAggregation": "sum", + "temporality": null, + "timeAggregation": "rate" + } + ], + "dataSource": "metrics", + "disabled": false, + "expression": "A", + "filter": { + "expression": "" + }, + "groupBy": [ + { + "key": "table", + "dataType": "string", + "type": "tag", + "isColumn": false + } + ], + "having": { + "expression": "" + }, + "legend": "{{table}}", + "orderBy": [], + "queryName": "A", + "stepInterval": 60 + } + ], + "queryFormulas": [] + }, + "clickhouse_sql": [ + { + "name": "A", + "query": "", + "legend": "", + "disabled": false + } + ], + "promql": [ + { + "name": "A", + "query": "", + "legend": "", + "disabled": false + } + ], + "id": "05b8bba7-a4d0-57b6-9cd7-d19eedc69171" + } + }, + { + "id": "8828b51f-f0fc-56b2-a674-21d95d230501", + "title": "HTTP — request / response body size p99 (bytes)", + "description": "", + "panelTypes": "graph", + "timePreferance": "GLOBAL_TIME", + "query": { + "queryType": "builder", + "builder": { + "queryData": [ + { + "aggregations": [ + { + "metricName": "http.server.request.body.size.bucket", + "reduceTo": "avg", + "spaceAggregation": "p99", + "temporality": null, + "timeAggregation": "rate" + } + ], + "dataSource": "metrics", + "disabled": false, + "expression": "A", + "filter": { + "expression": "" + }, + "groupBy": [], + "having": { + "expression": "" + }, + "legend": "request p99", + "orderBy": [], + "queryName": "A", + "stepInterval": 60 + }, + { + "aggregations": [ + { + "metricName": "http.server.response.body.size.bucket", + "reduceTo": "avg", + "spaceAggregation": "p99", + "temporality": null, + "timeAggregation": "rate" + } + ], + "dataSource": "metrics", + "disabled": false, + "expression": "B", + "filter": { + "expression": "" + }, + "groupBy": [], + "having": { + "expression": "" + }, + "legend": "response p99", + "orderBy": [], + "queryName": "B", + "stepInterval": 60 + } + ], + "queryFormulas": [] + }, + "clickhouse_sql": [ + { + "name": "A", + "query": "", + "legend": "", + "disabled": false + } + ], + "promql": [ + { + "name": "A", + "query": "", + "legend": "", + "disabled": false + } + ], + "id": "ed4527e0-8071-5d8f-9d80-a3d2139b6dfa" + }, + "yAxisUnit": "bytes" + } + ] +} diff --git a/deployments/signoz/load-dashboards.sh b/deployments/signoz/load-dashboards.sh new file mode 100755 index 00000000..882b72a9 --- /dev/null +++ b/deployments/signoz/load-dashboards.sh @@ -0,0 +1,76 @@ +#!/usr/bin/env bash +# +# Load (upsert) the WaveHouse SigNoz dashboards in ./dashboards/*.json into a +# running SigNoz instance. Existing dashboards are matched by title and updated +# in place (so bookmarked URLs survive); new ones are created. Re-run any time +# the JSON files change. +# +# SigNoz OSS has no on-disk dashboard provisioning (unlike Grafana) and disables +# self-registration, so this can't be folded into `docker compose up` — run it +# once after you've created your SigNoz account in the UI. +# +# Auth — provide ONE of: +# SIGNOZ_TOKEN= bearer token; in the SigNoz UI it's +# localStorage['AUTH_TOKEN'] +# SIGNOZ_EMAIL=you@example.com SIGNOZ_PASSWORD='...' logs in for you via /api/v1/login +# Optional: +# SIGNOZ_URL=http://localhost:3301 (default) +# +# Requires: curl, jq. +# +# Examples: +# SIGNOZ_EMAIL=you@example.com SIGNOZ_PASSWORD='hunter2' deployments/signoz/load-dashboards.sh +# SIGNOZ_TOKEN="$(pbpaste)" deployments/signoz/load-dashboards.sh +set -euo pipefail + +SIGNOZ_URL="${SIGNOZ_URL:-http://localhost:3301}" +DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/dashboards" + +command -v jq >/dev/null 2>&1 || { echo "error: jq is required (brew install jq | apt-get install jq)" >&2; exit 1; } +command -v curl >/dev/null 2>&1 || { echo "error: curl is required" >&2; exit 1; } + +# --- obtain a bearer token -------------------------------------------------- +TOKEN="${SIGNOZ_TOKEN:-}" +if [ -z "$TOKEN" ]; then + if [ -z "${SIGNOZ_EMAIL:-}" ] || [ -z "${SIGNOZ_PASSWORD:-}" ]; then + cat >&2 <&2; exit 1; } +fi +AUTH=(-H "Authorization: Bearer ${TOKEN}") + +# --- existing dashboards: title -> id --------------------------------------- +existing="$(curl -fsS "${AUTH[@]}" "${SIGNOZ_URL}/api/v1/dashboards")" + +shopt -s nullglob +files=("${DIR}"/*.json) +[ "${#files[@]}" -gt 0 ] || { echo "error: no dashboard JSON files in ${DIR}" >&2; exit 1; } + +for f in "${files[@]}"; do + title="$(jq -r '.title' "$f")" + id="$(printf '%s' "$existing" | jq -r --arg t "$title" 'first(.data[] | select(.data.title == $t) | .id) // empty')" + if [ -n "$id" ]; then + echo "↻ updating ${title} (${id})" + curl -fsS -o /dev/null -X PUT "${AUTH[@]}" -H 'Content-Type: application/json' \ + "${SIGNOZ_URL}/api/v1/dashboards/${id}" --data-binary @"$f" + else + echo "＋ creating ${title}" + curl -fsS -o /dev/null -X POST "${AUTH[@]}" -H 'Content-Type: application/json' \ + "${SIGNOZ_URL}/api/v1/dashboards" --data-binary @"$f" + fi +done + +echo "done — ${SIGNOZ_URL}/dashboard" diff --git a/docs/src/content/docs/deployment.md b/docs/src/content/docs/deployment.md index da72d56e..dd298546 100644 --- a/docs/src/content/docs/deployment.md +++ b/docs/src/content/docs/deployment.md @@ -381,6 +381,22 @@ docker compose -f deployments/signoz/docker-compose.yaml up -d The SigNoz UI is on `http://localhost:3301` (host `3301` → container `8080`; `8080` is left for WaveHouse). Point WaveHouse at the collector with `WH_OTEL_ADDR=127.0.0.1:4317` for a host-side WaveHouse (the default), or `WH_OTEL_ADDR=host.docker.internal:4317` from a WaveHouse container — the collector publishes `4317` on the host. See `deployments/compose/standalone.signoz.yaml` for a compose override that wires the containerized `standalone.yaml` WaveHouse into this stack. +### Dashboards + +Two dashboards live in `deployments/signoz/dashboards/` as version-controlled JSON — `wavehouse-overview.json` (HTTP traffic, latency, OTLP intake) and `wavehouse-runtime-internals.json` (Go runtime, embedded NATS, ingest pipeline, payload sizes). SigNoz OSS has no on-disk dashboard provisioning (unlike Grafana) and disables self-registration, so loading them is a one-time-per-person manual step rather than part of `compose up`: + +1. `docker compose -f deployments/signoz/docker-compose.yaml up -d` +2. Open `http://localhost:3301` and **create your account** (first visit only). +3. Load the dashboards: + + ```bash + SIGNOZ_EMAIL=you@example.com SIGNOZ_PASSWORD='…' deployments/signoz/load-dashboards.sh + ``` + +The loader upserts by title (matching dashboards are updated in place, new ones created), so re-run it whenever the JSON changes. It can also take a `SIGNOZ_TOKEN` (the `AUTH_TOKEN` from the SigNoz UI's `localStorage`) instead of email/password, and `SIGNOZ_URL` (default `http://localhost:3301`). Needs `curl` and `jq`. To edit a dashboard, change it in the UI, then re-export it: `GET /api/v1/dashboards/` and save the response's `.data.data` over the JSON file. + +The two **Ingest** panels (`wavehouse_bento_*`) are empty until events actually flow through the ingest pipeline — those counters aren't emitted until the first event is written. + ## Resetting Data in Development ### Option 1: Drop and Recreate Tables From ada727dbecccd4eea0cb2bdb329396a3a752f6c3 Mon Sep 17 00:00:00 2001 From: Jack Woods Date: Wed, 13 May 2026 13:48:37 -0400 Subject: [PATCH 04/25] feat(deploy): wire SigNoz into make dev workflow + harden compose stack MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Closes the remaining items from #121: integrate the SigNoz stack into the Makefile so `make dev-obs` is a one-shot "dev mode + SigNoz + WaveHouse pointed at OTLP collector" experience, and finish the stack hardening that PR #116's review surfaced. Makefile — new `##@ Observability` section mirroring the `deps-*` family: - signoz-up (idempotent, --wait on UI + collector health) - signoz-down (preserves UI history + admin account) - signoz-logs (tails UI + collector) - signoz-wipe (destroys volumes — admin account reset) - signoz-dashboards (upsert via load-dashboards.sh; guards on creds) - dev-obs (deps-up + signoz-up + air with WH_OTEL_ENABLED=true WH_OTEL_ADDR=127.0.0.1:4317; auto-loads dashboards iff SIGNOZ_TOKEN or SIGNOZ_EMAIL+PASSWORD in env, otherwise prints a pointer at signoz-dashboards) clean-all also tears down the SigNoz compose project so `make clean-all` is a true full reset. Compose hardening (deployments/signoz/compose.yaml — renamed from docker-compose.yaml per modern Compose v2 convention): - Drop `container_name:` from all six services so the project-prefix naming actually works (and multiple instances no longer collide). - Drop the explicit `signoz-net` network block — project-default works and `standalone.signoz.yaml` reaches the collector via host.docker.internal. - Add an `otel-collector` healthcheck against its `health_check` extension on :13133. The signoz/signoz-otel-collector image has no curl/wget but does have bash, so probe via /dev/tcp + GET / and grep for `200 OK`. This is what `--wait signoz otel-collector` blocks on. - Gate `signoz` and `otel-collector` on `signoz-telemetrystore-migrator: service_completed_successfully`. Without this, `--wait signoz otel-collector` skipped the migrator entirely (nothing pulled it into the closure) and the collector's `migrate sync check` retried forever against a nonexistent `signoz_traces` schema. Warm restarts hid the bug because the schema persisted in the volume — only a wipe + up surfaced it. - Header comment now points operators at `make signoz-up`, calls out the dev-only posture, and points at `deploy.resources.limits` as the prod knob. Docs: - docs/development.md — new "Running with observability" section with the target table + first-run admin-account guidance. - docs/deployment.md — refresh the SigNoz bring-up + dashboards flow around the new Make wrappers; update compose-path references. - README.md — quick-start bullet pointing at `make dev-obs`. WHissues.md, standalone.signoz.yaml: update `docker-compose.yaml` → `compose.yaml` path references. Co-Authored-By: Claude Opus 4.7 (1M context) --- Makefile | 67 +++++++++++++++++++ README.md | 2 + WHissues.md | 6 +- deployments/compose/standalone.signoz.yaml | 4 +- .../{docker-compose.yaml => compose.yaml} | 55 ++++++++++----- docs/src/content/docs/deployment.md | 11 +-- docs/src/content/docs/development.md | 44 ++++++++++++ 7 files changed, 164 insertions(+), 25 deletions(-) rename deployments/signoz/{docker-compose.yaml => compose.yaml} (75%) diff --git a/Makefile b/Makefile index 1f7b28bc..5d0cf5a0 100644 --- a/Makefile +++ b/Makefile @@ -228,6 +228,72 @@ deps-wipe: ## Stop ClickHouse AND destroy its data volume (DESTRUCTIVE — use t @echo "$(RED)==> Wiping ClickHouse (containers + volumes)...$(RESET)" @$(DEV_COMPOSE) down -v --remove-orphans +##@ Observability + +# SigNoz stack lives in its own compose file. Same DEV_COMPOSE pattern — +# alias once, keep recipes terse, easy to swap binaries (e.g. podman compose). +SIGNOZ_COMPOSE_FILE := deployments/signoz/compose.yaml +SIGNOZ_COMPOSE := docker compose -f $(SIGNOZ_COMPOSE_FILE) + +# `--wait` blocks on the named long-running services' healthchecks: +# signoz → /api/v1/health on :8080 +# otel-collector → health_check extension on :13133 +# clickhouse + zookeeper-1 reach healthy first via the dependency chain. +.PHONY: signoz-up +signoz-up: ## Start SigNoz stack (idempotent; blocks until UI + collector healthy) + @echo "$(CYAN)==> Starting SigNoz stack...$(RESET)" + @$(SIGNOZ_COMPOSE) up -d --wait signoz otel-collector + @echo " SigNoz UI: $(GREEN)http://localhost:3301$(RESET) (first-run: create local admin account)" + @echo " OTLP gRPC: $(GREEN)localhost:4317$(RESET)" + @echo " OTLP HTTP: $(GREEN)localhost:4318$(RESET)" + +.PHONY: signoz-down +signoz-down: ## Stop SigNoz stack (preserves volumes — UI history kept) + @echo "$(YELLOW)==> Stopping SigNoz...$(RESET)" + @$(SIGNOZ_COMPOSE) down + +.PHONY: signoz-logs +signoz-logs: ## Tail SigNoz UI + collector logs (Ctrl+C to detach) + @$(SIGNOZ_COMPOSE) logs -f signoz otel-collector + +.PHONY: signoz-wipe +signoz-wipe: ## Stop SigNoz AND destroy its volumes (DESTRUCTIVE — admin account reset) + @echo "$(RED)==> Wiping SigNoz (containers + volumes)...$(RESET)" + @$(SIGNOZ_COMPOSE) down -v --remove-orphans + +# Dashboard loader auths to the SigNoz API via either a JWT (SIGNOZ_TOKEN, +# pulled from the UI's localStorage AUTH_TOKEN) or email+password. Both are +# user-specific, so neither is committed — the guard fails fast with a +# pointer to the first-run flow rather than letting the loader emit a less +# obvious 401. +.PHONY: signoz-dashboards +signoz-dashboards: ## Upsert WaveHouse dashboards into local SigNoz (needs SIGNOZ_TOKEN or SIGNOZ_EMAIL+SIGNOZ_PASSWORD) + @if [ -z "$$SIGNOZ_TOKEN" ] && { [ -z "$$SIGNOZ_EMAIL" ] || [ -z "$$SIGNOZ_PASSWORD" ]; }; then \ + echo "$(RED)==> Set SIGNOZ_TOKEN, or SIGNOZ_EMAIL + SIGNOZ_PASSWORD.$(RESET)"; \ + echo " First-run: open http://localhost:3301 and create your admin account, then re-run."; \ + exit 1; \ + fi + @deployments/signoz/load-dashboards.sh + +# dev-obs = `make dev` + the SigNoz layer + WaveHouse pointed at the +# collector. `WH_OTEL_ENABLED` is the only flip needed; `WH_OTEL_ADDR` +# already defaults to 127.0.0.1:4317 in internal/config/config.go but is +# set explicitly so the recipe doubles as documentation. +.PHONY: dev-obs +dev-obs: deps-up signoz-up $(AIR) ## Hot-reload dev server with SigNoz observability + @echo "$(CYAN)==> Starting WaveHouse with SigNoz observability$(RESET)" + @echo " WaveHouse: $(GREEN)http://localhost:8080$(RESET)" + @echo " SigNoz UI: $(GREEN)http://localhost:3301$(RESET) (first-run: create local admin account)" + @if [ -n "$$SIGNOZ_TOKEN" ] || { [ -n "$$SIGNOZ_EMAIL" ] && [ -n "$$SIGNOZ_PASSWORD" ]; }; then \ + echo "$(CYAN)==> Loading dashboards (creds detected)...$(RESET)"; \ + deployments/signoz/load-dashboards.sh || echo "$(YELLOW)==> Dashboard load failed; continuing$(RESET)"; \ + else \ + echo " Dashboards: run $(CYAN)make signoz-dashboards$(RESET) after creating your SigNoz account"; \ + fi + @WH_OTEL_ENABLED=true \ + WH_OTEL_ADDR=127.0.0.1:4317 \ + $(AIR) -c .air.toml + ##@ Code Quality .PHONY: fmt @@ -571,6 +637,7 @@ clean-all: clean clean-test clean-tools ## Full reset — clean + clean-test + c @echo "$(YELLOW)==> Full reset (dev state, docker)...$(RESET)" @rm -rf data/ @$(DEV_COMPOSE) down -v --remove-orphans 2>/dev/null || true + @$(SIGNOZ_COMPOSE) down -v --remove-orphans 2>/dev/null || true @docker compose -f tests/e2e/compose.yaml down -v --remove-orphans 2>/dev/null || true ##@ Tooling diff --git a/README.md b/README.md index a437c934..edfb524a 100644 --- a/README.md +++ b/README.md @@ -134,6 +134,8 @@ make dev WaveHouse will automatically recompile and restart whenever you save a `.go` file. +**With observability:** `make dev-obs` is the same as `make dev`, plus a local [SigNoz](https://signoz.io/) stack on `http://localhost:3301` with WaveHouse already pointed at its OTLP collector. See [docs/development.md § Running with observability](docs/src/content/docs/development.md#running-with-observability--make-dev-obs). + ## 🤝 Contributing We welcome issues, pull requests, and feedback! Please see our [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines on how to structure your code and run the integration test suites. diff --git a/WHissues.md b/WHissues.md index 8a86588a..26ec2b8d 100644 --- a/WHissues.md +++ b/WHissues.md @@ -4,14 +4,14 @@ Running log of issues found while deploying/operating WaveHouse. Newest first. --- -## 1. SigNoz compose stack (`deployments/signoz/docker-compose.yaml`) fails to start — RESOLVED +## 1. SigNoz compose stack (`deployments/signoz/compose.yaml`) fails to start — RESOLVED **Found:** 2026-05-12, bringing the bundled SigNoz stack up locally on OrbStack. **Fixed:** 2026-05-12 — `deployments/signoz/` rebuilt from the upstream SigNoz `deploy/docker/docker-compose.yaml` (SigNoz `v0.122.0` / `signoz-otel-collector v0.144.3`), trimmed to a single node. -**Symptom (before):** `docker compose -f deployments/signoz/docker-compose.yaml up -d` +**Symptom (before):** `docker compose -f deployments/signoz/compose.yaml up -d` → `signoz-migrator` exits 1; consequently `signoz` (query-service) and `signoz-otel-collector` never start (both `depends_on: migrator: service_completed_successfully`). Only `signoz-clickhouse` stays up. @@ -128,7 +128,7 @@ query system.columns: code: 516, message: default: Authentication failed: passwo ``` **Cause:** `deployments/compose/standalone.yaml` and -`deployments/signoz/docker-compose.yaml` both define a service named `clickhouse`. +`deployments/signoz/compose.yaml` both define a service named `clickhouse`. On a container attached to both networks, the unqualified hostname `clickhouse` is ambiguous, and WaveHouse can end up dialing SigNoz's ClickHouse — which is configured with a non-empty password (`SIGNOZ_CH_PASSWORD`, default `password`), diff --git a/deployments/compose/standalone.signoz.yaml b/deployments/compose/standalone.signoz.yaml index e5551f0a..19643066 100644 --- a/deployments/compose/standalone.signoz.yaml +++ b/deployments/compose/standalone.signoz.yaml @@ -1,5 +1,5 @@ # Override for standalone.yaml that points WaveHouse's OTLP export at the -# SigNoz collector from deployments/signoz/docker-compose.yaml. +# SigNoz collector from deployments/signoz/compose.yaml. # # The SigNoz collector publishes 4317 on the host, so WaveHouse reaches it # via host.docker.internal — no shared Docker network. Joining the SigNoz @@ -9,7 +9,7 @@ # (non-empty password) instead of its own. # # Usage (bring the SigNoz stack up first): -# docker compose -f deployments/signoz/docker-compose.yaml up -d +# make signoz-up # or: docker compose -f deployments/signoz/compose.yaml up -d # docker compose -p wavehouse \ # -f deployments/compose/standalone.yaml \ # -f deployments/compose/standalone.signoz.yaml up -d --build diff --git a/deployments/signoz/docker-compose.yaml b/deployments/signoz/compose.yaml similarity index 75% rename from deployments/signoz/docker-compose.yaml rename to deployments/signoz/compose.yaml index b69f3cb3..fc407eb4 100644 --- a/deployments/signoz/docker-compose.yaml +++ b/deployments/signoz/compose.yaml @@ -3,8 +3,9 @@ # (SigNoz v0.122.0 / signoz-otel-collector v0.144.3), trimmed to a single node # and adapted to this repo's layout. # -# Bring it up: -# docker compose -f deployments/signoz/docker-compose.yaml up -d +# Bring it up (via the Makefile wrapper, or directly): +# make signoz-up +# docker compose -f deployments/signoz/compose.yaml up -d # # UI: http://localhost:3301 (host 3301 → container 8080; WaveHouse owns 8080) # OTLP gRPC: localhost:4317 (point WaveHouse at it: WH_OTEL_ADDR=127.0.0.1:4317, @@ -15,10 +16,12 @@ # One-shot init containers (normal to see them as `Exited (0)`): # - init-clickhouse: fetches the histogramQuantile UDF binary ClickHouse loads # - signoz-telemetrystore-migrator: creates the signoz_* ClickHouse schema, then exits +# +# Dev posture only — no resource limits set. For prod, add +# `deploy.resources.limits.{memory,cpus}` to clickhouse + signoz + +# otel-collector (the trace ingest path can be hungry under load). x-common: &common - networks: - - signoz-net restart: unless-stopped logging: options: @@ -57,7 +60,6 @@ services: init-clickhouse: <<: *common image: clickhouse/clickhouse-server:25.5.6 - container_name: signoz-init-clickhouse restart: on-failure command: - bash @@ -77,7 +79,6 @@ services: zookeeper-1: <<: *common image: signoz/zookeeper:3.7.1 - container_name: signoz-zookeeper-1 user: root environment: - ZOO_SERVER_ID=1 @@ -95,7 +96,6 @@ services: clickhouse: <<: *clickhouse-defaults - container_name: signoz-clickhouse # Ports are intentionally not published — this ClickHouse is internal to # SigNoz. (WaveHouse has its own ClickHouse in deployments/compose/.) volumes: @@ -107,9 +107,17 @@ services: - clickhouse:/var/lib/clickhouse/ signoz: - <<: *db-depend + <<: *common + # Wait for the migrator to complete — not just ClickHouse to be healthy. + # The signoz query service reads `signoz_*` tables that the migrator + # creates; on a cold start those tables don't exist yet, so without + # this dep the UI boots into a confused state. + depends_on: + clickhouse: + condition: service_healthy + signoz-telemetrystore-migrator: + condition: service_completed_successfully image: signoz/signoz:v0.122.0 - container_name: signoz ports: - "3301:8080" # UI + API (8080 in the container; published on 3301 — WaveHouse owns host 8080) volumes: @@ -126,9 +134,18 @@ services: retries: 3 otel-collector: - <<: *db-depend + <<: *common + # Wait for the migrator to complete. The collector's entrypoint runs + # `migrate sync check` before starting; on a cold start with no schema + # it loops forever ("Database signoz_traces does not exist") and never + # flips healthy — `--wait` then times out. The migrator is what creates + # signoz_traces in the first place. + depends_on: + clickhouse: + condition: service_healthy + signoz-telemetrystore-migrator: + condition: service_completed_successfully image: signoz/signoz-otel-collector:v0.144.3 - container_name: signoz-otel-collector entrypoint: ["/bin/sh"] command: - -c @@ -148,11 +165,21 @@ services: ports: - "4317:4317" # OTLP gRPC receiver — WaveHouse pushes here - "4318:4318" # OTLP HTTP receiver + # Probes the health_check extension declared in otel-collector-config.yaml + # (`extensions.health_check.endpoint: 0.0.0.0:13133`). The endpoint serves + # 200 OK once the pipeline is fully built and ready to accept telemetry. + # The signoz/signoz-otel-collector image ships without curl or wget, but + # has bash — so we hand-roll an HTTP probe via /dev/tcp and grep the + # status line. Port 13133 is internal (not published). + healthcheck: + test: ["CMD", "bash", "-c", "exec 3<>/dev/tcp/127.0.0.1/13133 && printf 'GET / HTTP/1.0\r\n\r\n' >&3 && grep -q '200 OK' <&3"] + interval: 30s + timeout: 5s + retries: 3 signoz-telemetrystore-migrator: <<: *db-depend image: signoz/signoz-otel-collector:v0.144.3 - container_name: signoz-telemetrystore-migrator restart: on-failure entrypoint: ["/bin/sh"] command: @@ -167,10 +194,6 @@ services: - SIGNOZ_OTEL_COLLECTOR_CLICKHOUSE_REPLICATION=true - SIGNOZ_OTEL_COLLECTOR_TIMEOUT=10m -networks: - signoz-net: - name: signoz-net - volumes: clickhouse: name: signoz-clickhouse diff --git a/docs/src/content/docs/deployment.md b/docs/src/content/docs/deployment.md index dd298546..1135c2e7 100644 --- a/docs/src/content/docs/deployment.md +++ b/docs/src/content/docs/deployment.md @@ -373,10 +373,11 @@ By default, `prometheus.port` is `0`, which mounts `/metrics` on the main API se For production posture where metrics should not be exposed on the public API listener, set `port` to a separate non-zero value (e.g. `9091`). WaveHouse spins up a dedicated HTTP listener bound to that port serving only `/metrics`. Firewall the port to internal networks only; the main API listener stays where it was. Both listeners participate in graceful shutdown. -`deployments/signoz/` is a self-contained Docker Compose setup for running SigNoz locally. It is modelled on the upstream SigNoz `deploy/docker/docker-compose.yaml` (SigNoz `v0.122.0` / `signoz-otel-collector v0.144.3`), trimmed to a single node: ClickHouse + ZooKeeper + the SigNoz query/UI service + the OTel collector (OTLP gRPC on `:4317`, HTTP on `:4318`), plus two one-shot init containers (`signoz-init-clickhouse` fetches the `histogramQuantile` UDF binary; `signoz-telemetrystore-migrator` creates the `signoz_*` ClickHouse schema). Both init containers exit `0` after running — that's expected. Bring it up: +`deployments/signoz/` is a self-contained Docker Compose setup for running SigNoz locally. It is modelled on the upstream SigNoz `deploy/docker/docker-compose.yaml` (SigNoz `v0.122.0` / `signoz-otel-collector v0.144.3`), trimmed to a single node: ClickHouse + ZooKeeper + the SigNoz query/UI service + the OTel collector (OTLP gRPC on `:4317`, HTTP on `:4318`), plus two one-shot init containers (`init-clickhouse` fetches the `histogramQuantile` UDF binary; `signoz-telemetrystore-migrator` creates the `signoz_*` ClickHouse schema). Both init containers exit `0` after running — that's expected. Bring it up via the Makefile wrapper (recommended) or compose directly: ```bash -docker compose -f deployments/signoz/docker-compose.yaml up -d +make signoz-up +# or: docker compose -f deployments/signoz/compose.yaml up -d ``` The SigNoz UI is on `http://localhost:3301` (host `3301` → container `8080`; `8080` is left for WaveHouse). Point WaveHouse at the collector with `WH_OTEL_ADDR=127.0.0.1:4317` for a host-side WaveHouse (the default), or `WH_OTEL_ADDR=host.docker.internal:4317` from a WaveHouse container — the collector publishes `4317` on the host. See `deployments/compose/standalone.signoz.yaml` for a compose override that wires the containerized `standalone.yaml` WaveHouse into this stack. @@ -385,11 +386,13 @@ The SigNoz UI is on `http://localhost:3301` (host `3301` → container `8080`; ` Two dashboards live in `deployments/signoz/dashboards/` as version-controlled JSON — `wavehouse-overview.json` (HTTP traffic, latency, OTLP intake) and `wavehouse-runtime-internals.json` (Go runtime, embedded NATS, ingest pipeline, payload sizes). SigNoz OSS has no on-disk dashboard provisioning (unlike Grafana) and disables self-registration, so loading them is a one-time-per-person manual step rather than part of `compose up`: -1. `docker compose -f deployments/signoz/docker-compose.yaml up -d` +1. `make signoz-up` (or `docker compose -f deployments/signoz/compose.yaml up -d`). 2. Open `http://localhost:3301` and **create your account** (first visit only). -3. Load the dashboards: +3. Load the dashboards (via Make wrapper or directly): ```bash + SIGNOZ_EMAIL=you@example.com SIGNOZ_PASSWORD='…' make signoz-dashboards + # or call the loader directly: SIGNOZ_EMAIL=you@example.com SIGNOZ_PASSWORD='…' deployments/signoz/load-dashboards.sh ``` diff --git a/docs/src/content/docs/development.md b/docs/src/content/docs/development.md index 9e3d8c25..c167cf42 100644 --- a/docs/src/content/docs/development.md +++ b/docs/src/content/docs/development.md @@ -151,6 +151,43 @@ when you need to poke at ClickHouse: **Stopping `make dev`**: `Ctrl+C` stops air, which propagates SIGINT to WaveHouse for a graceful shutdown (NATS JetStream flush, etc.). ClickHouse stays up — re-running `make dev` is fast because the volume is preserved. Use `make deps-down` or `make deps-wipe` to stop ClickHouse explicitly. +### Running with observability — `make dev-obs` + +`make dev-obs` is the dev-mode counterpart of `make dev`, with a local SigNoz stack alongside WaveHouse and the OTLP pipeline already pointed at it. Everything `make dev` gives you, plus traces / metrics / logs flowing into a UI on `http://localhost:3301`. + +```bash +make dev-obs +# WaveHouse: http://localhost:8080 +# SigNoz UI: http://localhost:3301 (first-run: create local admin account) +``` + +The stack is its own compose project under `deployments/signoz/compose.yaml` — ClickHouse + ZooKeeper + the SigNoz query/UI image + the OTel collector. `make dev-obs` brings up the SigNoz stack with `--wait` (blocks on the UI's `/api/v1/health` and the collector's `health_check` extension), then starts WaveHouse under air with `WH_OTEL_ENABLED=true` and `WH_OTEL_ADDR=127.0.0.1:4317`. + +**First run**: SigNoz prompts you to create a local admin account before any API request will authenticate — that's normal, not a misconfigured deployment. Once you've signed up, export credentials so subsequent runs auto-load the WaveHouse dashboards: + +```bash +export SIGNOZ_EMAIL=you@example.com +export SIGNOZ_PASSWORD='...' +# or grab AUTH_TOKEN from localStorage in the SigNoz UI: +# export SIGNOZ_TOKEN='...' +make dev-obs +``` + +When `SIGNOZ_TOKEN` or `SIGNOZ_EMAIL`+`SIGNOZ_PASSWORD` are set, `dev-obs` runs the dashboard loader best-effort (failure doesn't block the dev server). Without them, it prints a one-liner pointing at `make signoz-dashboards`. + +| Target | What it does | +| ------ | ------------ | +| `make signoz-up` | Start the SigNoz stack and block until UI + collector are healthy. Idempotent. | +| `make signoz-down` | Stop the stack. Volumes preserved — UI history and admin account stay. | +| `make signoz-logs` | `docker compose logs -f signoz otel-collector`. | +| `make signoz-wipe` | Stop **and** destroy volumes (admin account reset; UI prompts for signup again). | +| `make signoz-dashboards` | Upsert the WaveHouse dashboards into local SigNoz. Idempotent (upsert-by-title); needs `SIGNOZ_TOKEN` or `SIGNOZ_EMAIL`+`SIGNOZ_PASSWORD`. | +| `make dev-obs` | `make dev` + `make signoz-up` + WaveHouse pointed at the collector + (if creds set) `make signoz-dashboards`. | + +The two version-controlled dashboards live under `deployments/signoz/dashboards/`: `wavehouse-overview` (HTTP traffic, latency, OTLP intake) and `wavehouse-runtime-internals` (Go runtime, embedded NATS, ingest pipeline). To edit one, change it in the SigNoz UI, then re-export via the API: `GET /api/v1/dashboards/` and save the response's `.data.data` over the JSON file. + +The SigNoz `clickhouse` service is intentionally **separate** from WaveHouse's `clickhouse` (`deployments/compose/dependencies.yaml`). Don't dual-home a WaveHouse container onto both networks — the unqualified hostname becomes ambiguous; see `WHissues.md` §4. The dev-mode flow stays on `127.0.0.1:4317` and side-steps this entirely. + ### Using the SDK playground against `make dev` The `clients/ts/playground/` scripts (`public.ts`, `auth.ts`, `admin.ts`) target a WaveHouse with auth enabled in dev mode and the secret `sdk-dev-secret`. To match those defaults under `make dev`: @@ -412,6 +449,13 @@ Run `make help` to see all targets. Key ones: | `make deps-logs` | Tail ClickHouse logs | | `make deps-shell` | `clickhouse-client` REPL on the running container | | `make deps-wipe` | Stop ClickHouse AND destroy its data volume (DESTRUCTIVE) | +| **Observability** | | +| `make dev-obs` | `make dev` + local SigNoz stack with WaveHouse already pointed at its OTLP collector | +| `make signoz-up` | Start SigNoz stack alone (idempotent; blocks until healthy) | +| `make signoz-down` | Stop SigNoz stack (preserves UI history + admin account) | +| `make signoz-logs` | Tail SigNoz UI + collector logs | +| `make signoz-wipe` | Stop SigNoz AND destroy its volumes (DESTRUCTIVE — admin account reset) | +| `make signoz-dashboards` | Upsert version-controlled dashboards into local SigNoz | | **Static checks** | | | `make fmt` | Check formatting (run `make fix` to apply) | | `make tidy` | Verify `go.mod`/`go.sum` are tidy (run `make fix` to apply) | From 764eb6f9af4fc40f0398278d68089c1f9924a952 Mon Sep 17 00:00:00 2001 From: Jack Woods Date: Wed, 13 May 2026 15:01:47 -0400 Subject: [PATCH 05/25] fix(deploy): SigNoz dashboard loader is token-only MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit v0.122.0 moved password login from /api/v1/login to /api/v2/sessions/email_password and now requires an org UUID that isn't externally discoverable, so the SIGNOZ_EMAIL+SIGNOZ_PASSWORD branch silently dead-ended. Drop it and require SIGNOZ_TOKEN (the JWT the SPA already keeps in localStorage AUTH_TOKEN) — one DevTools click away. Update the script header, Makefile guard, dev-obs autoload check, and dashboards/README.md to match. Co-Authored-By: Claude Opus 4.7 (1M context) --- Makefile | 23 +++++++------- deployments/signoz/dashboards/README.md | 6 ++-- deployments/signoz/load-dashboards.sh | 40 ++++++++++--------------- 3 files changed, 31 insertions(+), 38 deletions(-) diff --git a/Makefile b/Makefile index 5d0cf5a0..4e69125f 100644 --- a/Makefile +++ b/Makefile @@ -261,16 +261,17 @@ signoz-wipe: ## Stop SigNoz AND destroy its volumes (DESTRUCTIVE — admin accou @echo "$(RED)==> Wiping SigNoz (containers + volumes)...$(RESET)" @$(SIGNOZ_COMPOSE) down -v --remove-orphans -# Dashboard loader auths to the SigNoz API via either a JWT (SIGNOZ_TOKEN, -# pulled from the UI's localStorage AUTH_TOKEN) or email+password. Both are -# user-specific, so neither is committed — the guard fails fast with a -# pointer to the first-run flow rather than letting the loader emit a less -# obvious 401. +# Dashboard loader auths to the SigNoz API with a JWT (SIGNOZ_TOKEN, pulled +# from the UI's localStorage AUTH_TOKEN). v0.122.0 moved password login to an +# endpoint that needs an org UUID we can't externally discover, so the token +# is the only supported path. The guard fails fast with a pointer to the +# first-run flow rather than letting the loader emit a less obvious 401. .PHONY: signoz-dashboards -signoz-dashboards: ## Upsert WaveHouse dashboards into local SigNoz (needs SIGNOZ_TOKEN or SIGNOZ_EMAIL+SIGNOZ_PASSWORD) - @if [ -z "$$SIGNOZ_TOKEN" ] && { [ -z "$$SIGNOZ_EMAIL" ] || [ -z "$$SIGNOZ_PASSWORD" ]; }; then \ - echo "$(RED)==> Set SIGNOZ_TOKEN, or SIGNOZ_EMAIL + SIGNOZ_PASSWORD.$(RESET)"; \ - echo " First-run: open http://localhost:3301 and create your admin account, then re-run."; \ +signoz-dashboards: ## Upsert WaveHouse dashboards into local SigNoz (needs SIGNOZ_TOKEN) + @if [ -z "$$SIGNOZ_TOKEN" ]; then \ + echo "$(RED)==> Set SIGNOZ_TOKEN.$(RESET)"; \ + echo " First-run: open http://localhost:3301, create your admin account,"; \ + echo " then copy AUTH_TOKEN from DevTools → Application → Local Storage."; \ exit 1; \ fi @deployments/signoz/load-dashboards.sh @@ -284,8 +285,8 @@ dev-obs: deps-up signoz-up $(AIR) ## Hot-reload dev server with SigNoz observabi @echo "$(CYAN)==> Starting WaveHouse with SigNoz observability$(RESET)" @echo " WaveHouse: $(GREEN)http://localhost:8080$(RESET)" @echo " SigNoz UI: $(GREEN)http://localhost:3301$(RESET) (first-run: create local admin account)" - @if [ -n "$$SIGNOZ_TOKEN" ] || { [ -n "$$SIGNOZ_EMAIL" ] && [ -n "$$SIGNOZ_PASSWORD" ]; }; then \ - echo "$(CYAN)==> Loading dashboards (creds detected)...$(RESET)"; \ + @if [ -n "$$SIGNOZ_TOKEN" ]; then \ + echo "$(CYAN)==> Loading dashboards (SIGNOZ_TOKEN detected)...$(RESET)"; \ deployments/signoz/load-dashboards.sh || echo "$(YELLOW)==> Dashboard load failed; continuing$(RESET)"; \ else \ echo " Dashboards: run $(CYAN)make signoz-dashboards$(RESET) after creating your SigNoz account"; \ diff --git a/deployments/signoz/dashboards/README.md b/deployments/signoz/dashboards/README.md index a0699858..fdf525c5 100644 --- a/deployments/signoz/dashboards/README.md +++ b/deployments/signoz/dashboards/README.md @@ -11,13 +11,13 @@ Format: SigNoz query-builder **v5** schema (the `version` field). Each file is e ## Loading them -SigNoz OSS doesn't provision dashboards from disk, so after the stack is up and you've created your account in the UI: +SigNoz OSS doesn't provision dashboards from disk, so after the stack is up and you've created your account in the UI, grab the `AUTH_TOKEN` from DevTools → Application → Local Storage → `http://localhost:3301`, then: ```bash -SIGNOZ_EMAIL=you@example.com SIGNOZ_PASSWORD='…' ../load-dashboards.sh +SIGNOZ_TOKEN='eyJ…' ../load-dashboards.sh ``` -`../load-dashboards.sh` upserts by title (existing dashboards are updated in place), so re-run it whenever these files change. See its header for auth options (`SIGNOZ_TOKEN`, `SIGNOZ_URL`). +`../load-dashboards.sh` upserts by title (existing dashboards are updated in place), so re-run it whenever these files change. See its header for the full token-grab walkthrough and `SIGNOZ_URL` overrides. ## Editing a dashboard diff --git a/deployments/signoz/load-dashboards.sh b/deployments/signoz/load-dashboards.sh index 882b72a9..59b81389 100755 --- a/deployments/signoz/load-dashboards.sh +++ b/deployments/signoz/load-dashboards.sh @@ -9,18 +9,20 @@ # self-registration, so this can't be folded into `docker compose up` — run it # once after you've created your SigNoz account in the UI. # -# Auth — provide ONE of: -# SIGNOZ_TOKEN= bearer token; in the SigNoz UI it's -# localStorage['AUTH_TOKEN'] -# SIGNOZ_EMAIL=you@example.com SIGNOZ_PASSWORD='...' logs in for you via /api/v1/login +# Auth: SIGNOZ_TOKEN (the JWT used by the SPA itself). +# 1. Sign in at ${SIGNOZ_URL} (default http://localhost:3301). +# 2. DevTools → Application → Local Storage → http://localhost:3301 → copy +# the value of `AUTH_TOKEN`. +# 3. Export it: `export SIGNOZ_TOKEN='eyJ...'`. +# +# (SigNoz v0.122.0 moved username/password login to `/api/v2/sessions/email_password` +# and requires an org UUID that isn't externally discoverable, so we no longer +# offer email/password login here — the token is one click away.) +# # Optional: # SIGNOZ_URL=http://localhost:3301 (default) # # Requires: curl, jq. -# -# Examples: -# SIGNOZ_EMAIL=you@example.com SIGNOZ_PASSWORD='hunter2' deployments/signoz/load-dashboards.sh -# SIGNOZ_TOKEN="$(pbpaste)" deployments/signoz/load-dashboards.sh set -euo pipefail SIGNOZ_URL="${SIGNOZ_URL:-http://localhost:3301}" @@ -29,26 +31,16 @@ DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/dashboards" command -v jq >/dev/null 2>&1 || { echo "error: jq is required (brew install jq | apt-get install jq)" >&2; exit 1; } command -v curl >/dev/null 2>&1 || { echo "error: curl is required" >&2; exit 1; } -# --- obtain a bearer token -------------------------------------------------- TOKEN="${SIGNOZ_TOKEN:-}" if [ -z "$TOKEN" ]; then - if [ -z "${SIGNOZ_EMAIL:-}" ] || [ -z "${SIGNOZ_PASSWORD:-}" ]; then - cat >&2 <&2 <&2; exit 1; } + exit 1 fi AUTH=(-H "Authorization: Bearer ${TOKEN}") From 189dc77c2cd583dd7cc4d5fdf56a9cb8cb4396a8 Mon Sep 17 00:00:00 2001 From: Jack Woods Date: Wed, 13 May 2026 15:47:00 -0400 Subject: [PATCH 06/25] test(config): mark RejectsMalformedHeaders subtest parallel The parent test calls t.Parallel(), so tparallel requires the t.Run closure to do the same or the lint step fails. Co-Authored-By: Claude Opus 4.7 (1M context) --- internal/config/config_test.go | 1 + 1 file changed, 1 insertion(+) diff --git a/internal/config/config_test.go b/internal/config/config_test.go index 3783b9ad..92658ccc 100644 --- a/internal/config/config_test.go +++ b/internal/config/config_test.go @@ -328,6 +328,7 @@ func TestValidate_RejectsMalformedHeaders(t *testing.T) { } for _, tc := range cases { t.Run(tc.name, func(t *testing.T) { + t.Parallel() cfg := Config{ Server: Server{Port: 8080}, ClickHouse: ClickHouse{HTTPScheme: "http"}, From f22ed374a58edb6fec3eaeb2819d8125de7535c8 Mon Sep 17 00:00:00 2001 From: Jack Woods <113313601+jfwoods@users.noreply.github.com> Date: Wed, 13 May 2026 16:50:01 -0400 Subject: [PATCH 07/25] Update docs/src/content/docs/deployment.md Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> --- docs/src/content/docs/deployment.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/src/content/docs/deployment.md b/docs/src/content/docs/deployment.md index 1135c2e7..9b3dd24e 100644 --- a/docs/src/content/docs/deployment.md +++ b/docs/src/content/docs/deployment.md @@ -390,11 +390,11 @@ Two dashboards live in `deployments/signoz/dashboards/` as version-controlled JS 2. Open `http://localhost:3301` and **create your account** (first visit only). 3. Load the dashboards (via Make wrapper or directly): - ```bash - SIGNOZ_EMAIL=you@example.com SIGNOZ_PASSWORD='…' make signoz-dashboards - # or call the loader directly: - SIGNOZ_EMAIL=you@example.com SIGNOZ_PASSWORD='…' deployments/signoz/load-dashboards.sh - ``` +# First, get the token from the SigNoz UI's localStorage +export SIGNOZ_TOKEN='eyJ...' +make signoz-dashboards +# or call the loader directly: +deployments/signoz/load-dashboards.sh The loader upserts by title (matching dashboards are updated in place, new ones created), so re-run it whenever the JSON changes. It can also take a `SIGNOZ_TOKEN` (the `AUTH_TOKEN` from the SigNoz UI's `localStorage`) instead of email/password, and `SIGNOZ_URL` (default `http://localhost:3301`). Needs `curl` and `jq`. To edit a dashboard, change it in the UI, then re-export it: `GET /api/v1/dashboards/` and save the response's `.data.data` over the JSON file. From 96a430ddcd9795838c183406235766a908833fad Mon Sep 17 00:00:00 2001 From: Jack Woods <113313601+jfwoods@users.noreply.github.com> Date: Wed, 13 May 2026 16:53:34 -0400 Subject: [PATCH 08/25] Update docs/src/content/docs/development.md Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> --- docs/src/content/docs/development.md | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/docs/src/content/docs/development.md b/docs/src/content/docs/development.md index c167cf42..fa6041e7 100644 --- a/docs/src/content/docs/development.md +++ b/docs/src/content/docs/development.md @@ -166,10 +166,8 @@ The stack is its own compose project under `deployments/signoz/compose.yaml` — **First run**: SigNoz prompts you to create a local admin account before any API request will authenticate — that's normal, not a misconfigured deployment. Once you've signed up, export credentials so subsequent runs auto-load the WaveHouse dashboards: ```bash -export SIGNOZ_EMAIL=you@example.com -export SIGNOZ_PASSWORD='...' -# or grab AUTH_TOKEN from localStorage in the SigNoz UI: -# export SIGNOZ_TOKEN='...' +# Grab AUTH_TOKEN from localStorage in the SigNoz UI: +export SIGNOZ_TOKEN='...' make dev-obs ``` From df4d6c462bd1dff4551f6bc0046071115433c0ad Mon Sep 17 00:00:00 2001 From: Jack Woods <113313601+jfwoods@users.noreply.github.com> Date: Wed, 13 May 2026 16:54:24 -0400 Subject: [PATCH 09/25] Update docs/src/content/docs/development.md Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> --- docs/src/content/docs/development.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/src/content/docs/development.md b/docs/src/content/docs/development.md index fa6041e7..e01bc254 100644 --- a/docs/src/content/docs/development.md +++ b/docs/src/content/docs/development.md @@ -179,7 +179,7 @@ When `SIGNOZ_TOKEN` or `SIGNOZ_EMAIL`+`SIGNOZ_PASSWORD` are set, `dev-obs` runs | `make signoz-down` | Stop the stack. Volumes preserved — UI history and admin account stay. | | `make signoz-logs` | `docker compose logs -f signoz otel-collector`. | | `make signoz-wipe` | Stop **and** destroy volumes (admin account reset; UI prompts for signup again). | -| `make signoz-dashboards` | Upsert the WaveHouse dashboards into local SigNoz. Idempotent (upsert-by-title); needs `SIGNOZ_TOKEN` or `SIGNOZ_EMAIL`+`SIGNOZ_PASSWORD`. | +| make signoz-dashboards | Upsert the WaveHouse dashboards into local SigNoz. Idempotent (upsert-by-title); needs SIGNOZ_TOKEN. | | `make dev-obs` | `make dev` + `make signoz-up` + WaveHouse pointed at the collector + (if creds set) `make signoz-dashboards`. | The two version-controlled dashboards live under `deployments/signoz/dashboards/`: `wavehouse-overview` (HTTP traffic, latency, OTLP intake) and `wavehouse-runtime-internals` (Go runtime, embedded NATS, ingest pipeline). To edit one, change it in the SigNoz UI, then re-export via the API: `GET /api/v1/dashboards/` and save the response's `.data.data` over the JSON file. From eceb045c185fd8cc412184e05798c0109eb07664 Mon Sep 17 00:00:00 2001 From: Jack Woods <113313601+jfwoods@users.noreply.github.com> Date: Wed, 13 May 2026 16:59:14 -0400 Subject: [PATCH 10/25] Update docs/src/content/docs/deployment.md Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com> --- docs/src/content/docs/deployment.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/src/content/docs/deployment.md b/docs/src/content/docs/deployment.md index 9b3dd24e..38fe48c0 100644 --- a/docs/src/content/docs/deployment.md +++ b/docs/src/content/docs/deployment.md @@ -343,7 +343,7 @@ export WH_OTEL_ENABLED=true export WH_OTEL_ADDR=https://otlp-gateway-prod-us-east-0.grafana.net:443 # instanceID:token, base64-encoded; backslash-escape the space if your shell needs it export WH_OTEL_HEADERS=authorization=Basic $(printf '%s' "$INSTANCE_ID:$TOKEN" | base64) -``` +export WH_OTEL_HEADERS=authorization=Basic $(printf '%s' "$INSTANCE_ID:$TOKEN" | base64 | tr -d '\n') **Datadog OTLP intake:** From 9681288f743e0ababe339af5e873b42525563d1c Mon Sep 17 00:00:00 2001 From: Jack Woods <113313601+jfwoods@users.noreply.github.com> Date: Wed, 13 May 2026 17:05:03 -0400 Subject: [PATCH 11/25] Update deployments/signoz/compose.yaml Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> --- deployments/signoz/compose.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/deployments/signoz/compose.yaml b/deployments/signoz/compose.yaml index fc407eb4..a0fad53c 100644 --- a/deployments/signoz/compose.yaml +++ b/deployments/signoz/compose.yaml @@ -126,7 +126,7 @@ services: - SIGNOZ_ALERTMANAGER_PROVIDER=signoz - SIGNOZ_TELEMETRYSTORE_CLICKHOUSE_DSN=tcp://clickhouse:9000 - SIGNOZ_SQLSTORE_SQLITE_PATH=/var/lib/signoz/signoz.db - - SIGNOZ_TOKENIZER_JWT_SECRET=secret + - SIGNOZ_TOKENIZER_JWT_SECRET=${SIGNOZ_TOKENIZER_JWT_SECRET:?set SIGNOZ_TOKENIZER_JWT_SECRET} healthcheck: test: ["CMD", "wget", "--spider", "-q", "localhost:8080/api/v1/health"] interval: 30s From 2cb7248936a222f03e456b710558b1292d59628f Mon Sep 17 00:00:00 2001 From: Jack Woods <113313601+jfwoods@users.noreply.github.com> Date: Wed, 13 May 2026 17:07:48 -0400 Subject: [PATCH 12/25] Update deployments/signoz/load-dashboards.sh Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> --- deployments/signoz/load-dashboards.sh | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/deployments/signoz/load-dashboards.sh b/deployments/signoz/load-dashboards.sh index 59b81389..2f1a0304 100755 --- a/deployments/signoz/load-dashboards.sh +++ b/deployments/signoz/load-dashboards.sh @@ -53,6 +53,10 @@ files=("${DIR}"/*.json) for f in "${files[@]}"; do title="$(jq -r '.title' "$f")" + if [ -z "$title" ] || [ "$title" = "null" ]; then + echo "error: dashboard file '$f' must contain a non-empty .title" >&2 + exit 1 + fi id="$(printf '%s' "$existing" | jq -r --arg t "$title" 'first(.data[] | select(.data.title == $t) | .id) // empty')" if [ -n "$id" ]; then echo "↻ updating ${title} (${id})" From 7d61933f385e0a51ee663cb610b64167b4d13243 Mon Sep 17 00:00:00 2001 From: Jack Woods Date: Wed, 13 May 2026 17:12:21 -0400 Subject: [PATCH 13/25] fix(deploy): scope SigNoz overview widgets to wavehouse service Empty filter expressions caused the overview dashboard to aggregate telemetry across every service reporting to the SigNoz instance. Restrict the HTTP and span widgets to service.name = 'wavehouse'; leave the collector-intake widget unfiltered since its metrics (otelcol_receiver_accepted_*) are emitted by the collector itself. Co-Authored-By: Claude Opus 4.7 (1M context) --- .../signoz/dashboards/wavehouse-overview.json | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/deployments/signoz/dashboards/wavehouse-overview.json b/deployments/signoz/dashboards/wavehouse-overview.json index c02f573f..97b44b71 100644 --- a/deployments/signoz/dashboards/wavehouse-overview.json +++ b/deployments/signoz/dashboards/wavehouse-overview.json @@ -75,7 +75,7 @@ "disabled": false, "expression": "A", "filter": { - "expression": "" + "expression": "service.name = 'wavehouse'" }, "groupBy": [], "having": { @@ -133,7 +133,7 @@ "disabled": false, "expression": "A", "filter": { - "expression": "" + "expression": "service.name = 'wavehouse'" }, "groupBy": [ { @@ -198,7 +198,7 @@ "disabled": false, "expression": "A", "filter": { - "expression": "" + "expression": "service.name = 'wavehouse'" }, "groupBy": [], "having": { @@ -223,7 +223,7 @@ "disabled": false, "expression": "B", "filter": { - "expression": "" + "expression": "service.name = 'wavehouse'" }, "groupBy": [], "having": { @@ -248,7 +248,7 @@ "disabled": false, "expression": "C", "filter": { - "expression": "" + "expression": "service.name = 'wavehouse'" }, "groupBy": [], "having": { @@ -306,7 +306,7 @@ "disabled": false, "expression": "A", "filter": { - "expression": "" + "expression": "service.name = 'wavehouse'" }, "groupBy": [ { @@ -371,7 +371,7 @@ "disabled": false, "expression": "A", "filter": { - "expression": "" + "expression": "service.name = 'wavehouse'" }, "groupBy": [ { From e8172a0ec413f570b7d654f6e6a6318dcc23a01f Mon Sep 17 00:00:00 2001 From: Jack Woods <113313601+jfwoods@users.noreply.github.com> Date: Wed, 13 May 2026 17:31:16 -0400 Subject: [PATCH 14/25] Update deployments/signoz/compose.yaml Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com> --- deployments/signoz/compose.yaml | 1 + 1 file changed, 1 insertion(+) diff --git a/deployments/signoz/compose.yaml b/deployments/signoz/compose.yaml index a0fad53c..2a231268 100644 --- a/deployments/signoz/compose.yaml +++ b/deployments/signoz/compose.yaml @@ -71,6 +71,7 @@ services: echo "Fetching histogram-quantile binary for $${node_os}/$${node_arch}" cd /tmp wget -O histogram-quantile.tar.gz "https://github.com/SigNoz/signoz/releases/download/histogram-quantile%2F$${version}/histogram-quantile_$${node_os}_$${node_arch}.tar.gz" + echo " histogram-quantile.tar.gz" | sha256sum -c - tar -xvzf histogram-quantile.tar.gz mv histogram-quantile /var/lib/clickhouse/user_scripts/histogramQuantile volumes: From ccfb9474406ab5e4a3a95983d2df590bd53da7a7 Mon Sep 17 00:00:00 2001 From: Jack Woods Date: Wed, 13 May 2026 21:51:54 -0400 Subject: [PATCH 15/25] =?UTF-8?q?docs:=20address=20PR=20#135=20review=20th?= =?UTF-8?q?reads=20=E2=80=94=20Datadog=20rework=20+=20token-only=20loader?= =?UTF-8?q?=20+=20lint?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Fixes the [MUST] and unresolved CodeRabbit comments on this PR's doc surface. deployment.md — three coupled issues in the OTLP patterns section: - Datadog example was structurally wrong (CodeRabbit Critical): `WH_OTEL_ADDR=https://otlp.datadoghq.com:4317` was a public-Datadog hostname that doesn't accept OTLP, on the local Agent port, with the wrong scheme for a plaintext gRPC receiver. Datadog publishes no direct-to-cloud OTLP endpoint — the supported path is the DDOT Collector (Datadog Distribution of OpenTelemetry Collector) embedded in the Agent, GA for k8s / Preview for Linux, which exposes a standard OTLP receiver on 4317/4318 and forwards via the bundled datadogexporter. Reworked: dropped "Datadog OTLP" from the "Direct-to-cloud" pattern title (Honeycomb + Grafana Cloud are still truly direct), added a new "Pattern: Datadog (via local DDOT Collector)" subsection with host + k8s downward-API examples, links to the DDOT install guide, and a note about the legacy "OTLP-Ingest-in-the-Agent" path compatibility. - The Grafana Cloud `base64` line was malformed: a partial fix had been applied that left both the broken and the corrected `export` lines side-by-side and never closed the bash fence (so the entire Datadog block downstream was rendering inside that code block). Removed the duplicate, dropped the confusing "backslash-escape the space" comment per CodeRabbit's suggestion, kept the `| tr -d '\n'` version with a clearer comment, and re-balanced the fence. - Dashboards-section loader snippet (lines starting with `#`) was rendering as H1 headings (markdownlint MD022 / CodeRabbit). Wrapped in a bash fenced block under list item 3. - "It can also take a SIGNOZ_TOKEN ... instead of email/password" was a leftover from the pre-token-only revision and inverted reality (the token is the only path). Rewrote to "It requires SIGNOZ_TOKEN ... and accepts SIGNOZ_URL ...". - "disables self-registration" terminology was ambiguous (SigNoz OSS doesn't disable user signup; the relevant fact is no on-disk dashboard provisioning). Replaced with "does not support dashboard provisioning from disk". development.md — same email/password cleanup: - "When SIGNOZ_TOKEN or SIGNOZ_EMAIL+SIGNOZ_PASSWORD are set, dev-obs ..." → "When SIGNOZ_TOKEN is set, dev-obs ..." (the Makefile recipe only ever checks the token). - signoz-dashboards table row now has the same backtick formatting as every other row in the table. - dev-obs row's "(if creds set)" → "(if SIGNOZ_TOKEN set)" for consistency with the prose above. WHissues.md — markdownlint MD040: added a `text` language tag to the four opening fences that were missing one (the migrator-bootstrap, mq-init, schema-discovery, and ClickHouse-auth error blocks). The `... ON CLUSTER ` trailing-space comment was dismissed upstream as extraneous and is left alone. --- WHissues.md | 8 ++--- docs/src/content/docs/deployment.md | 54 ++++++++++++++++++++-------- docs/src/content/docs/development.md | 6 ++-- 3 files changed, 46 insertions(+), 22 deletions(-) diff --git a/WHissues.md b/WHissues.md index 26ec2b8d..15fa0f26 100644 --- a/WHissues.md +++ b/WHissues.md @@ -18,7 +18,7 @@ service_completed_successfully`). Only `signoz-clickhouse` stays up. Migrator log: -``` +```text migrate/bootstrap.go:119 Creating databases Error: failed to bootstrap store for migrations: code: 62, message: Syntax error: failed at position 42 (end of query): . Expected one of: identifier, string literal ``` @@ -75,7 +75,7 @@ steady (data flows regardless). **Symptom:** `docker compose -f deployments/compose/standalone.yaml up -d --build` → the `wavehouse` container exits 1 on boot: -``` +```text mq init failed error="create stream: nats: API error: code=500 err_code=10047 description=insufficient storage resources available" path=/app/data/nats ``` @@ -98,7 +98,7 @@ init failure as fatal. **Symptom:** on a cold `up`, the `wavehouse` container exits 1: -``` +```text schema discovery failed on boot error="query system.columns: dial tcp :9000: connect: connection refused" ``` @@ -123,7 +123,7 @@ the `wavehouse` container after ClickHouse is up. **Symptom:** if you put the WaveHouse container on the SigNoz compose network so it can reach `signoz-otel-collector:4317` by name, schema discovery fails with: -``` +```text query system.columns: code: 516, message: default: Authentication failed: password is incorrect, or there is no user with such name. ``` diff --git a/docs/src/content/docs/deployment.md b/docs/src/content/docs/deployment.md index 38fe48c0..44e91a54 100644 --- a/docs/src/content/docs/deployment.md +++ b/docs/src/content/docs/deployment.md @@ -324,7 +324,7 @@ otel: addr: 127.0.0.1:4317 # bare host:port — plaintext gRPC ``` -### Pattern: Direct-to-cloud OTLP (Honeycomb, Grafana Cloud, Datadog OTLP, etc.) +### Pattern: Direct-to-cloud OTLP (Honeycomb, Grafana Cloud) `otel.addr` is scheme-aware: `https://` selects TLS, `http://` or no scheme stays plaintext. Combine with `otel.headers` for the per-RPC auth that every cloud OTLP gateway expects. No sidecar required — a sidecar is still useful for egress queuing, batching, and tail-based sampling, but is no longer the only way to get TLS + auth. @@ -341,21 +341,43 @@ export WH_OTEL_HEADERS=x-honeycomb-team=YOUR_API_KEY ```bash export WH_OTEL_ENABLED=true export WH_OTEL_ADDR=https://otlp-gateway-prod-us-east-0.grafana.net:443 -# instanceID:token, base64-encoded; backslash-escape the space if your shell needs it -export WH_OTEL_HEADERS=authorization=Basic $(printf '%s' "$INSTANCE_ID:$TOKEN" | base64) +# instanceID:token, base64-encoded (tr -d '\n' strips base64's 76-char line wrap) export WH_OTEL_HEADERS=authorization=Basic $(printf '%s' "$INSTANCE_ID:$TOKEN" | base64 | tr -d '\n') +``` + +If different signals need different gateway hosts (Grafana Cloud's traces/metrics/logs endpoints differ in some regions), set `WH_OTEL_TRACES_ADDR` / `WH_OTEL_METRICS_ADDR` / `WH_OTEL_LOGS_ADDR` to override the default per signal. Empty means inherit from `otel.addr`. Headers always apply to every exporter — there is intentionally no per-signal header override. + +mTLS / client-certificate auth is not yet supported; open an issue if you need it. + +### Pattern: Datadog (via local DDOT Collector) + +Datadog has no public direct-to-cloud OTLP endpoint — telemetry must transit a local OTLP receiver that re-exports to Datadog over Datadog's own protocol. The recommended receiver is the [DDOT Collector](https://docs.datadoghq.com/opentelemetry/setup/ddot_collector/) (Datadog Distribution of OpenTelemetry Collector), embedded in the Datadog Agent — GA on Kubernetes, Preview on Linux. It exposes a standard OTLP receiver on `4317` (gRPC) and `4318` (HTTP) and forwards to Datadog via the bundled `datadogexporter`. Point WaveHouse at the local receiver — the API-key auth lives on the Agent, not on the OTLP hop, so `WH_OTEL_HEADERS` stays empty. -**Datadog OTLP intake:** +**Host (DDOT Collector on the same host as WaveHouse):** ```bash export WH_OTEL_ENABLED=true -export WH_OTEL_ADDR=https://otlp.datadoghq.com:4317 -export WH_OTEL_HEADERS=dd-api-key=YOUR_API_KEY +export WH_OTEL_ADDR=127.0.0.1:4317 # plaintext gRPC; no scheme +# WH_OTEL_HEADERS intentionally unset — DD_API_KEY is on the Agent ``` -If different signals need different gateway hosts (Grafana Cloud's traces/metrics/logs endpoints differ in some regions), set `WH_OTEL_TRACES_ADDR` / `WH_OTEL_METRICS_ADDR` / `WH_OTEL_LOGS_ADDR` to override the default per signal. Empty means inherit from `otel.addr`. Headers always apply to every exporter — there is intentionally no per-signal header override. +**Kubernetes (DDOT Collector via the Datadog Operator or Helm chart):** -mTLS / client-certificate auth is not yet supported; open an issue if you need it. +WaveHouse reaches the Agent on the same node via the downward API: + +```yaml +env: + - name: HOST_IP + valueFrom: + fieldRef: + fieldPath: status.hostIP + - name: WH_OTEL_ENABLED + value: "true" + - name: WH_OTEL_ADDR + value: "$(HOST_IP):4317" +``` + +See Datadog's [DDOT install guide](https://docs.datadoghq.com/opentelemetry/setup/ddot_collector/install/kubernetes/) for enabling the OTLP receiver on the Operator/Helm side. The legacy [OTLP Ingest in the Agent](https://docs.datadoghq.com/opentelemetry/setup/otlp_ingest_in_the_agent/) path (Agent ≥ 6.32 / 7.32) exposes the same receiver shape and works identically from WaveHouse's side — DDOT is the recommended replacement. ### Pattern: Grafana Cloud / Mimir / Loki / Tempo via Grafana Alloy @@ -384,19 +406,21 @@ The SigNoz UI is on `http://localhost:3301` (host `3301` → container `8080`; ` ### Dashboards -Two dashboards live in `deployments/signoz/dashboards/` as version-controlled JSON — `wavehouse-overview.json` (HTTP traffic, latency, OTLP intake) and `wavehouse-runtime-internals.json` (Go runtime, embedded NATS, ingest pipeline, payload sizes). SigNoz OSS has no on-disk dashboard provisioning (unlike Grafana) and disables self-registration, so loading them is a one-time-per-person manual step rather than part of `compose up`: +Two dashboards live in `deployments/signoz/dashboards/` as version-controlled JSON — `wavehouse-overview.json` (HTTP traffic, latency, OTLP intake) and `wavehouse-runtime-internals.json` (Go runtime, embedded NATS, ingest pipeline, payload sizes). SigNoz OSS does not support dashboard provisioning from disk (unlike Grafana), so loading them is a one-time-per-person manual step rather than part of `compose up`: 1. `make signoz-up` (or `docker compose -f deployments/signoz/compose.yaml up -d`). 2. Open `http://localhost:3301` and **create your account** (first visit only). 3. Load the dashboards (via Make wrapper or directly): -# First, get the token from the SigNoz UI's localStorage -export SIGNOZ_TOKEN='eyJ...' -make signoz-dashboards -# or call the loader directly: -deployments/signoz/load-dashboards.sh + ```bash + # First, get the token from the SigNoz UI's localStorage + export SIGNOZ_TOKEN='eyJ...' + make signoz-dashboards + # or call the loader directly: + deployments/signoz/load-dashboards.sh + ``` -The loader upserts by title (matching dashboards are updated in place, new ones created), so re-run it whenever the JSON changes. It can also take a `SIGNOZ_TOKEN` (the `AUTH_TOKEN` from the SigNoz UI's `localStorage`) instead of email/password, and `SIGNOZ_URL` (default `http://localhost:3301`). Needs `curl` and `jq`. To edit a dashboard, change it in the UI, then re-export it: `GET /api/v1/dashboards/` and save the response's `.data.data` over the JSON file. +The loader upserts by title (matching dashboards are updated in place, new ones created), so re-run it whenever the JSON changes. It requires `SIGNOZ_TOKEN` (the `AUTH_TOKEN` from the SigNoz UI's `localStorage`) and accepts `SIGNOZ_URL` (default `http://localhost:3301`). Needs `curl` and `jq`. To edit a dashboard, change it in the UI, then re-export it: `GET /api/v1/dashboards/` and save the response's `.data.data` over the JSON file. The two **Ingest** panels (`wavehouse_bento_*`) are empty until events actually flow through the ingest pipeline — those counters aren't emitted until the first event is written. diff --git a/docs/src/content/docs/development.md b/docs/src/content/docs/development.md index e01bc254..da210f56 100644 --- a/docs/src/content/docs/development.md +++ b/docs/src/content/docs/development.md @@ -171,7 +171,7 @@ export SIGNOZ_TOKEN='...' make dev-obs ``` -When `SIGNOZ_TOKEN` or `SIGNOZ_EMAIL`+`SIGNOZ_PASSWORD` are set, `dev-obs` runs the dashboard loader best-effort (failure doesn't block the dev server). Without them, it prints a one-liner pointing at `make signoz-dashboards`. +When `SIGNOZ_TOKEN` is set, `dev-obs` runs the dashboard loader best-effort (failure doesn't block the dev server). Without it, `dev-obs` prints a one-liner pointing at `make signoz-dashboards`. | Target | What it does | | ------ | ------------ | @@ -179,8 +179,8 @@ When `SIGNOZ_TOKEN` or `SIGNOZ_EMAIL`+`SIGNOZ_PASSWORD` are set, `dev-obs` runs | `make signoz-down` | Stop the stack. Volumes preserved — UI history and admin account stay. | | `make signoz-logs` | `docker compose logs -f signoz otel-collector`. | | `make signoz-wipe` | Stop **and** destroy volumes (admin account reset; UI prompts for signup again). | -| make signoz-dashboards | Upsert the WaveHouse dashboards into local SigNoz. Idempotent (upsert-by-title); needs SIGNOZ_TOKEN. | -| `make dev-obs` | `make dev` + `make signoz-up` + WaveHouse pointed at the collector + (if creds set) `make signoz-dashboards`. | +| `make signoz-dashboards` | Upsert the WaveHouse dashboards into local SigNoz. Idempotent (upsert-by-title); needs `SIGNOZ_TOKEN`. | +| `make dev-obs` | `make dev` + `make signoz-up` + WaveHouse pointed at the collector + (if `SIGNOZ_TOKEN` set) `make signoz-dashboards`. | The two version-controlled dashboards live under `deployments/signoz/dashboards/`: `wavehouse-overview` (HTTP traffic, latency, OTLP intake) and `wavehouse-runtime-internals` (Go runtime, embedded NATS, ingest pipeline). To edit one, change it in the SigNoz UI, then re-export via the API: `GET /api/v1/dashboards/` and save the response's `.data.data` over the JSON file. From 42b06b132ab6f9707d818247f88e73f6449f84a6 Mon Sep 17 00:00:00 2001 From: Jack Woods Date: Wed, 13 May 2026 21:52:06 -0400 Subject: [PATCH 16/25] test(observability): use testify assertions in endpoint_test MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Drops `reflect.DeepEqual` + `t.Fatalf` in favor of `require.Equal` / `require.Error` / `require.NoError` to match the project convention documented under AGENTS.md `**/*_test.go` coding guideline. Test coverage and behavior unchanged — verified via `go test ./internal/observability/... -run 'TestParseEndpoint|TestParseOTelHeaders' -count=1 -v`. --- internal/observability/endpoint_test.go | 17 +++++++---------- 1 file changed, 7 insertions(+), 10 deletions(-) diff --git a/internal/observability/endpoint_test.go b/internal/observability/endpoint_test.go index b59da76c..2a65737b 100644 --- a/internal/observability/endpoint_test.go +++ b/internal/observability/endpoint_test.go @@ -1,8 +1,9 @@ package observability import ( - "reflect" "testing" + + "github.com/stretchr/testify/require" ) func TestParseEndpoint(t *testing.T) { @@ -23,9 +24,8 @@ func TestParseEndpoint(t *testing.T) { for _, tc := range cases { t.Run(tc.name, func(t *testing.T) { gotHost, gotTLS := ParseEndpoint(tc.in) - if gotHost != tc.wantHost || gotTLS != tc.wantTLS { - t.Fatalf("ParseEndpoint(%q) = (%q, %v); want (%q, %v)", tc.in, gotHost, gotTLS, tc.wantHost, tc.wantTLS) - } + require.Equal(t, tc.wantHost, gotHost) + require.Equal(t, tc.wantTLS, gotTLS) }) } } @@ -51,15 +51,12 @@ func TestParseOTelHeaders(t *testing.T) { for _, tc := range cases { t.Run(tc.name, func(t *testing.T) { got, err := ParseOTelHeaders(tc.in) - if (err != nil) != tc.wantErr { - t.Fatalf("ParseOTelHeaders(%q) err = %v; wantErr = %v", tc.in, err, tc.wantErr) - } if tc.wantErr { + require.Error(t, err) return } - if !reflect.DeepEqual(got, tc.want) { - t.Fatalf("ParseOTelHeaders(%q) = %v; want %v", tc.in, got, tc.want) - } + require.NoError(t, err) + require.Equal(t, tc.want, got) }) } } From a92c90e5eeaa3cf073bb0793f85dff2319fc5919 Mon Sep 17 00:00:00 2001 From: Jack Woods Date: Wed, 13 May 2026 21:52:17 -0400 Subject: [PATCH 17/25] docs(changelog): SigNoz local dev stack entry + Datadog clarification MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds an `Unreleased / Added` bullet covering the local SigNoz dev stack rebuild (closes #121): version-pinned compose mirror of upstream `v0.122.0`, dashboards, token-only `load-dashboards.sh`, the `signoz-up/down/logs/wipe/dashboards` and `dev-obs` Make targets, `standalone.signoz.yaml` override, and the gotchas tracked in `WHissues.md`. Calls out token-only auth inline (SigNoz `v0.122.0` moved password login to `/api/v2/sessions/email_password` with an otherwise-undiscoverable org UUID, so the email/password branch dead- ended silently). Also corrects the existing direct-to-cloud OTLP bullet, which claimed the feature unlocked direct push to Datadog "without a sidecar." Datadog publishes no direct-to-cloud OTLP endpoint — the supported path is the DDOT Collector embedded in the Datadog Agent, which from WaveHouse's side is just a local plaintext OTLP receiver on `127.0.0.1:4317`. Pointer to the worked example in `docs/deployment.md`. --- CHANGELOG.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index ae675cdc..6b9ad767 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,7 +17,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - **Dependabot auto-merge no longer slipped past CI on consolidated-pipeline PRs** (`main branch protection` ruleset; `.github/workflows/project-orchestrator.yml`; `.github/workflows/claude-review.yml`; `AGENTS.md`): on commit `93b3206` the previously six-job CI was collapsed into a single job named `CI`, but three places still referenced the old multi-job names — the ruleset's `required_status_checks` (`Validate` + `Admin approval` only — `CI` not listed), the `bot-clean` check in `project-orchestrator.yml` (looking for `Build`/`Validate`/`Lint`/`Test`/`Integration Tests`/`SDK Tests`), and `REQUIRED_CHECKS` in `claude-review.yml` (same six). Net effect: a Dependabot PR opened at T+0 would have its `Validate` and `Admin approval` checks satisfied within ~30s (PR-title workflow + the auto-approval workflow), `gh pr merge --auto` would fire, GitHub would see all *required* checks green, and the PR would squash-merge before `ci.yml` even started running. Pre-consolidation runs (PR #90 etc.) didn't expose this because GitHub's auto-merge waits for in-flight checks as a courtesy, and the multi-job CI was producing checks before auto-merge fired — but with one collapsed `CI` check that didn't start until later, the courtesy wait disappeared. Fix: added `CI` to the ruleset's required-checks (now `Admin approval` / `CI` / `Validate`) via the `gh api PUT /repos/Wave-RF/WaveHouse/rulesets/15353356` round-trip, and updated both bot-clean lists to look for `CI` + `Validate`. AGENTS.md note about the required-check set updated to match. Diagnosis steps recorded inline as code comments in both workflow files so future-me knows where to look if the check name changes again. ### Added -- **Direct-to-cloud OTLP — TLS + auth headers, no sidecar required** (`internal/config/config.go`, `internal/observability/provider.go`, `internal/observability/endpoint.go`, `internal/testutil/otlp.go`, `cmd/wavehouse/main.go`, `tests/integration/otel_test.go`, `docs/src/content/docs/{configuration,deployment}.md`, `AGENTS.md`): closes [#97](https://github.com/Wave-RF/WaveHouse/issues/97). `otel.addr` is now scheme-aware — `https://` selects TLS (system root CAs by default), `http://` or a bare `host:port` stays plaintext gRPC (backward-compat with the prior `WithInsecure()` default). New `otel.headers` (`WH_OTEL_HEADERS`) takes the OTel spec env-var format — comma-separated `key=value`, values may contain `=` so base64 padding round-trips — and applies the parsed map as gRPC metadata to every OTLP exporter (traces, metrics, logs). Together these unlock direct push to Honeycomb / Grafana Cloud / Datadog OTLP without a sidecar. Per-signal endpoint overrides (`otel.{traces,metrics,logs}.addr`) point one signal at a different gateway than the top-level default (Grafana Cloud uses distinct trace/metric/log hosts in some regions); empty means inherit. Headers always apply uniformly — there is intentionally no per-signal header override. Test surface: `FakeOTLP` gains a `NewFakeOTLPTLS(t)` constructor (ephemeral self-signed cert + matching client `*tls.Config`) and per-signal `LastTraceHeaders / LastMetricHeaders / LastLogHeaders` accessors via captured gRPC metadata. New integration tests pin the TLS dial path, headers-on-all-signals propagation, and per-signal endpoint splitting. Validation rejects malformed `otel.headers` (missing `=`, empty key) at boot rather than letting a typo silently disable auth in production. mTLS / client-certificate auth is not yet supported. +- **Local SigNoz dev stack + Makefile integration + version-controlled dashboards** (`deployments/signoz/compose.yaml`, `deployments/signoz/clickhouse/{config,users,cluster,custom-function}.xml`, `deployments/signoz/otel-collector-opamp-config.yaml`, `deployments/signoz/dashboards/{wavehouse-overview,wavehouse-runtime-internals}.json`, `deployments/signoz/dashboards/README.md`, `deployments/signoz/load-dashboards.sh`, `deployments/compose/standalone.signoz.yaml`, `Makefile`, `WHissues.md`, `docs/src/content/docs/{deployment,development}.md`, `README.md`): closes [#121](https://github.com/Wave-RF/WaveHouse/issues/121). Replaces the hand-rolled `deployments/signoz/docker-compose.yaml` (pinned `:latest` everywhere and broke against the current `signoz-otel-collector` migrate CLI — see `WHissues.md` §1) with a version-pinned single-node mirror of upstream SigNoz `v0.122.0`'s `deploy/docker/docker-compose.yaml`: `signoz/signoz:v0.122.0` (consolidated UI/API on `:3301`) + `signoz/signoz-otel-collector:v0.144.3` (OTLP gRPC `:4317` / HTTP `:4318`) + dedicated `signoz-telemetrystore-migrator` (gates `signoz` and `otel-collector` on `service_completed_successfully`) + `clickhouse-server:25.5.6` + `zookeeper:3.7.1` + one-shot `init-clickhouse` (fetches the `histogramQuantile` UDF). Healthchecks added on the collector via `/dev/tcp` + `:13133/health_check` so `--wait` is meaningful. Headers comment now points operators at `make signoz-up`. Compose `container_name:` dropped throughout so the project-prefix naming works and multiple instances don't collide. New `##@ Observability` Make targets — `signoz-up` (idempotent, `--wait` on UI + collector), `signoz-down` (preserves admin account), `signoz-logs`, `signoz-wipe` (volumes + admin reset), `signoz-dashboards` (upserts via `load-dashboards.sh`; fail-fast on missing `SIGNOZ_TOKEN`), and `dev-obs` (`deps-up` + `signoz-up` + `air` with `WH_OTEL_ENABLED=true WH_OTEL_ADDR=127.0.0.1:4317`; auto-loads dashboards when `SIGNOZ_TOKEN` is in env, otherwise prints a pointer). `clean-all` now tears down the SigNoz project too. Two dashboards ship as version-controlled JSON — `wavehouse-overview` (HTTP traffic / latency / OTLP intake; widgets scoped to `service.name=wavehouse` so they don't aggregate other services that may also report to a local SigNoz) and `wavehouse-runtime-internals` (Go runtime, embedded NATS, ingest pipeline, payload sizes). `load-dashboards.sh` upserts by title against the running stack; auth is **token-only** because SigNoz `v0.122.0` moved password login to `/api/v2/sessions/email_password` and requires an org UUID that isn't externally discoverable, so the legacy `SIGNOZ_EMAIL`+`SIGNOZ_PASSWORD` branch dead-ended silently — the JWT the SPA keeps in `localStorage` under `AUTH_TOKEN` is one DevTools click away and is what `SIGNOZ_TOKEN` now wants. `standalone.signoz.yaml` wires the containerized `standalone.yaml` WaveHouse into the SigNoz stack via `host.docker.internal:4317`, caps `WH_MQ_MAX_BYTES_GB` for small dev disks (`WHissues.md` §2), and gates WaveHouse on a ClickHouse healthcheck (§3). Don't dual-home a WaveHouse container onto both SigNoz and `standalone.yaml` ClickHouse networks (§4 — name resolution collision; the `host.docker.internal` route sidesteps this). +- **Direct-to-cloud OTLP — TLS + auth headers, no sidecar required** (`internal/config/config.go`, `internal/observability/provider.go`, `internal/observability/endpoint.go`, `internal/testutil/otlp.go`, `cmd/wavehouse/main.go`, `tests/integration/otel_test.go`, `docs/src/content/docs/{configuration,deployment}.md`, `AGENTS.md`): closes [#97](https://github.com/Wave-RF/WaveHouse/issues/97). `otel.addr` is now scheme-aware — `https://` selects TLS (system root CAs by default), `http://` or a bare `host:port` stays plaintext gRPC (backward-compat with the prior `WithInsecure()` default). New `otel.headers` (`WH_OTEL_HEADERS`) takes the OTel spec env-var format — comma-separated `key=value`, values may contain `=` so base64 padding round-trips — and applies the parsed map as gRPC metadata to every OTLP exporter (traces, metrics, logs). Together these unlock direct push to Honeycomb and Grafana Cloud's OTLP gateway without a sidecar. Datadog does not publish a direct-to-cloud OTLP endpoint — its supported path remains the DDOT Collector embedded in the Datadog Agent (host or k8s), which WaveHouse reaches as a plaintext local OTLP receiver on `127.0.0.1:4317` (no headers — the API-key auth lives on the Agent); see `docs/deployment.md` for the worked example. Per-signal endpoint overrides (`otel.{traces,metrics,logs}.addr`) point one signal at a different gateway than the top-level default (Grafana Cloud uses distinct trace/metric/log hosts in some regions); empty means inherit. Headers always apply uniformly — there is intentionally no per-signal header override. Test surface: `FakeOTLP` gains a `NewFakeOTLPTLS(t)` constructor (ephemeral self-signed cert + matching client `*tls.Config`) and per-signal `LastTraceHeaders / LastMetricHeaders / LastLogHeaders` accessors via captured gRPC metadata. New integration tests pin the TLS dial path, headers-on-all-signals propagation, and per-signal endpoint splitting. Validation rejects malformed `otel.headers` (missing `=`, empty key) at boot rather than letting a typo silently disable auth in production. mTLS / client-certificate auth is not yet supported. - **Astro / Starlight documentation site at `wavehouse.dev`** (`docs/`, `Makefile`, `.github/dependabot.yml`, `.github/workflows/ci.yml`, `.gitignore`, `.vscode/launch.json`): full content + tooling for the public docs (Getting Started, Why WaveHouse, Architecture, API Reference, TypeScript SDK, Configuration, Deployment, Development). Build-time mermaid via `rehype-mermaid` (inline-svg strategy, no client-side JS), LaTeX via `remark-math` + `rehype-katex`, image zoom via `starlight-image-zoom`, expressiveCode with github dark/light + `env`/`dns` Shiki language aliases. PostHog frontend telemetry (key is public by design — embedded in every visitor's browser). `editLink` pointed at `main`; `lastUpdated` on. Sidebar lives in `docs/src/config/sidebar.ts` as the single source of truth, consumed by both Starlight rendering and the LLM-friendly outputs below. Cloudflare Workers + Static Assets deployment via `docs/wrangler.jsonc` (no auto-deploy yet — wired up when the GH Actions workflow lands separately). Make targets stay minimal: `dev-docs` (Astro dev server on :4321), `build-docs`, `preview-docs` — the last serves the production build through `wrangler dev`, so the Cloudflare Worker's content negotiation (`Accept: text/markdown` / LLM-bot User-Agent → `.md` twin) is exercised end-to-end the same way it will be in production. - **Per-page `.md` twin + `llms.txt` family via `starlight-llm-tools`** (`docs/astro.config.mjs`, `docs/worker/index.ts`, `docs/wrangler.jsonc`): every doc is also served as raw markdown at `.md` with a navigation header (Section / Subpages / Related / HTML version pointers), plus three concatenated views — `/llms.txt` manifest, `/llms-full.txt` (every page, sidebar-ordered), `/llms-small.txt` (overview pages only). Page-title slot gains a **Copy Markdown** button and **Open with AI** dropdown (Claude / ChatGPT / Cursor). Content negotiation is handled by a Cloudflare Worker that re-exports the `cloudflare-md-router` package's default handler in one line — `Accept: text/markdown` or known LLM-bot User-Agent (GPTBot, ClaudeBot, PerplexityBot, Applebot-Extended, etc.) gets the `.md` twin transparently, with a fall-through to the HTML response when the twin doesn't exist. - **Two extracted plugin packages, MIT-licensed**: [`github.com/Wave-RF/cloudflare-md-router`](https://github.com/Wave-RF/cloudflare-md-router) (Workers handler + `createMdRouter()` factory + extensible bot-UA regex; consumed in `docs/worker/index.ts` as a one-line re-export) and [`github.com/Wave-RF/starlight-llm-tools`](https://github.com/Wave-RF/starlight-llm-tools) (Starlight plugin that auto-injects the four routes, the two components, and the `PageTitle` override; optionally calls into `starlight-glossary/transform` when present). Both pinned via `github:Wave-RF/...` in `docs/package.json`; lockfile pins the resolved commit SHAs. Pre-npm-publish state — switch to normal version specifiers when the packages land on npm. From 380daf50ee331cfdab01b530688f37c2565d97ca Mon Sep 17 00:00:00 2001 From: Jack Woods Date: Wed, 13 May 2026 21:54:06 -0400 Subject: [PATCH 18/25] fix(deploy): SIGNOZ_TOKENIZER_JWT_SECRET no longer hard-fails fresh checkouts MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The earlier rotation to `${SIGNOZ_TOKENIZER_JWT_SECRET:?...}` (commit 6abb1ce, addressing CodeRabbit's "don't hardcode a JWT secret" feedback) was overcorrected for what this compose file actually is: a local-dev SigNoz stack, not a production deployment. The `:?` syntax made `docker compose up` (and therefore `make signoz-up` and `make dev-obs`) fail on every fresh clone with `required variable SIGNOZ_TOKENIZER_JWT_SECRET is not set` — no working setup, no `.env.example` left over (it was deleted earlier in the PR), and no docs mentioning the new variable. Switched to `:-wavehouse-local-dev-only` so the stack boots out of the box, with an inline comment pointing operators repurposing the file at the `openssl rand -hex 32` override pattern. The header already calls out "Dev posture only" so the default is consistent with the file's overall stance. Closes the two `[MUST]` review threads on compose.yaml:128 and :130. --- deployments/signoz/compose.yaml | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/deployments/signoz/compose.yaml b/deployments/signoz/compose.yaml index 2a231268..333e892c 100644 --- a/deployments/signoz/compose.yaml +++ b/deployments/signoz/compose.yaml @@ -127,7 +127,10 @@ services: - SIGNOZ_ALERTMANAGER_PROVIDER=signoz - SIGNOZ_TELEMETRYSTORE_CLICKHOUSE_DSN=tcp://clickhouse:9000 - SIGNOZ_SQLSTORE_SQLITE_PATH=/var/lib/signoz/signoz.db - - SIGNOZ_TOKENIZER_JWT_SECRET=${SIGNOZ_TOKENIZER_JWT_SECRET:?set SIGNOZ_TOKENIZER_JWT_SECRET} + # Dev-only default — this is a local-dev compose. Override with `export + # SIGNOZ_TOKENIZER_JWT_SECRET=$(openssl rand -hex 32)` before `make + # signoz-up` if you're repurposing this stack for anything non-local. + - SIGNOZ_TOKENIZER_JWT_SECRET=${SIGNOZ_TOKENIZER_JWT_SECRET:-wavehouse-local-dev-only} healthcheck: test: ["CMD", "wget", "--spider", "-q", "localhost:8080/api/v1/health"] interval: 30s From a9a6eb75b06cfa01741a9f4fe4f7d094e1614681 Mon Sep 17 00:00:00 2001 From: Jack Woods Date: Wed, 13 May 2026 22:10:01 -0400 Subject: [PATCH 19/25] fix(deploy): real histogram-quantile checksums + config.yaml otel parity MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two re-posted [MUST] threads from the recent claude[bot] pass. deployments/signoz/compose.yaml — histogram-quantile integrity check was inert: the previous block had `` as a literal placeholder, so `sha256sum -c -` exited non-zero on every run, and because the block had no `set -e` the failure was silently swallowed and `tar`/`mv` proceeded against whatever bytes wget happened to fetch (including HTML error pages on a transient outage). The earlier CodeRabbit "verify integrity" finding (commit 845b808 → da9f24a) only wired the suggested form without a real hash; it was theater rather than a check. Replaced with: - `set -euo pipefail` at the top so wget failures, partial downloads, or a missing case branch all abort cleanly. - A `case "$node_os_$node_arch"` block with the real SHA256s for all four platform/arch combos SigNoz ships (linux/amd64, linux/arm64, darwin/amd64, darwin/arm64), verified against upstream's signed checksums file at https://github.com/SigNoz/signoz/releases/download/histogram-quantile/v0.0.1/histogram-quantile_0.0.1_checksums.txt - Inline comment naming the checksums file URL so the next bump (`version=v0.0.2+`) has a clear remediation path. - Unknown platforms now `exit 1` with a clear message instead of silently downloading without verification. config.yaml — `otel:` block was missing the four fields added in afae85f: `otel.headers` and per-signal `otel.{traces,metrics,logs}.addr` overrides. AGENTS.md §"Documentation & Consistency Sync" rule 10 ("Go struct tags in config.go → Must match in config.yaml") wasn't satisfied. Added all four as commented-out lines (defaults are empty) with the same explanatory comments as the rest of the block. Closes the two re-posted [MUST] threads at compose.yaml:74 and config.go:51. --- config.yaml | 6 +++++- deployments/signoz/compose.yaml | 17 ++++++++++++++--- 2 files changed, 19 insertions(+), 4 deletions(-) diff --git a/config.yaml b/config.yaml index 38b0b1f7..a3c07b1e 100644 --- a/config.yaml +++ b/config.yaml @@ -19,14 +19,18 @@ server: otel: enabled: false # master switch — set true to export via OTLP gRPC - addr: "127.0.0.1:4317" + addr: "127.0.0.1:4317" # `https://...` → TLS, `http://...` or bare host:port → plaintext + # headers: "" # comma-separated key=value, e.g. "x-honeycomb-team=API_KEY" traces: enabled: true + # addr: "" # per-signal endpoint override; empty inherits otel.addr sample_rate: 1.0 # head-based, [0.0, 1.0]; tune down for high QPS metrics: enabled: true # OTLP push for metrics + # addr: "" # per-signal endpoint override; empty inherits otel.addr logs: enabled: true + # addr: "" # per-signal endpoint override; empty inherits otel.addr sample_rate: 1.0 # DEBUG/INFO OTLP rate; WARN+ always 100%, stdout always 100% # Prometheus exposition is independent of OTel — works on its own (Alloy / diff --git a/deployments/signoz/compose.yaml b/deployments/signoz/compose.yaml index 333e892c..96f1202b 100644 --- a/deployments/signoz/compose.yaml +++ b/deployments/signoz/compose.yaml @@ -65,14 +65,25 @@ services: - bash - -c - | + set -euo pipefail + # SHA256s from upstream's signed checksums file: + # https://github.com/SigNoz/signoz/releases/download/histogram-quantile/v0.0.1/histogram-quantile_0.0.1_checksums.txt + # Update both `version` and the case branches together when bumping. version="v0.0.1" node_os=$$(uname -s | tr '[:upper:]' '[:lower:]') node_arch=$$(uname -m | sed s/aarch64/arm64/ | sed s/x86_64/amd64/) + case "$${node_os}_$${node_arch}" in + linux_amd64) expected_sha=33997073eb6d82b7be4f27f9fc8ec9e28a395e0f20674d5d54bb2fa28a75488d ;; + linux_arm64) expected_sha=e5605ebffa82a450ebbcdf6cf19dad546e1e40d52dbf3e03cfd2d2d5b7394211 ;; + darwin_amd64) expected_sha=5ec1043e57a089380d5fac5fef6e078124d6b825737831ef05f382a17687a626 ;; + darwin_arm64) expected_sha=4e6d87f723a141472582f6ef04a935c61870b05ed94d518dc7c29f3d5e9db038 ;; + *) echo "no histogram-quantile SHA256 pinned for $${node_os}/$${node_arch}" >&2; exit 1 ;; + esac echo "Fetching histogram-quantile binary for $${node_os}/$${node_arch}" cd /tmp - wget -O histogram-quantile.tar.gz "https://github.com/SigNoz/signoz/releases/download/histogram-quantile%2F$${version}/histogram-quantile_$${node_os}_$${node_arch}.tar.gz" - echo " histogram-quantile.tar.gz" | sha256sum -c - - tar -xvzf histogram-quantile.tar.gz + wget -q -O histogram-quantile.tar.gz "https://github.com/SigNoz/signoz/releases/download/histogram-quantile%2F$${version}/histogram-quantile_$${node_os}_$${node_arch}.tar.gz" + echo "$${expected_sha} histogram-quantile.tar.gz" | sha256sum -c - + tar -xzf histogram-quantile.tar.gz mv histogram-quantile /var/lib/clickhouse/user_scripts/histogramQuantile volumes: - clickhouse-scripts:/var/lib/clickhouse/user_scripts/ From 98621af7e5ebf5c354ca7993ab06e53f92426204 Mon Sep 17 00:00:00 2001 From: Jack Woods Date: Wed, 13 May 2026 22:49:06 -0400 Subject: [PATCH 20/25] fix(observability): refuse to start if otel.headers parse fails post-validation MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `cmd/wavehouse/main.go` previously discarded the error from `observability.ParseOTelHeaders` with a comment noting that the parser cannot fail for input that `Validate()` already accepted. The assumption holds today — `internal/config/config.go`'s `validateOTelHeaders` is hand-mirrored from the same parsing logic — but the two implementations are independent. If they ever drift (e.g. a future auth-scheme tweak lands in one but not the other), `validateOTelHeaders` would accept input that `ParseOTelHeaders` silently truncates, the OTLP exporters would ship with no auth metadata, and the failure would surface only as rejected-on-the-wire telemetry — not as a startup error or a log message. Switched the blank-identifier to a real error check that logs and returns 1. Matches the fatal-error pattern used everywhere else in `run()`. Costs four lines; eliminates the silent-auth-drop class. Possible follow-up (not in this commit): consolidate the two parsers into a single leaf package both `config` and `observability` import, so drift is impossible by construction. Out of scope here because it shifts the package dependency graph (config currently imports nothing from observability so the rest of the codebase doesn't transitively pull in the OTel SDK through config). Closes the security-MEDIUM [MUST] thread at cmd/wavehouse/main.go:107. --- cmd/wavehouse/main.go | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-) diff --git a/cmd/wavehouse/main.go b/cmd/wavehouse/main.go index 90fab7cf..8d50a5dc 100644 --- a/cmd/wavehouse/main.go +++ b/cmd/wavehouse/main.go @@ -102,9 +102,17 @@ func run() int { // wanted — Prometheus-only operation (Alloy/scrape, no collector) is a // first-class mode. The OTel SDK MeterProvider is the shared substrate. if cfg.OTel.Enabled || cfg.Prometheus.Enabled { - // Headers already passed config validation; the parser cannot fail - // here for input that satisfied Validate(). - headers, _ := observability.ParseOTelHeaders(cfg.OTel.Headers) + // Validate() ran validateOTelHeaders, which mirrors ParseOTelHeaders — + // in practice the parse always succeeds here. The defensive check + // exists so that any future drift between the two implementations + // surfaces loudly instead of silently shipping OTLP exporters with no + // auth metadata (which would only show up as rejected-on-the-wire + // telemetry, not as a startup error). + headers, err := observability.ParseOTelHeaders(cfg.OTel.Headers) + if err != nil { + logger.Error("otel.headers parse failed after Validate accepted them; refusing to start with bad auth config", "error", err) + return 1 + } otelShutdown, ph, err := observability.InitProvider(ctx, serviceName, observability.ProviderConfig{ Endpoint: cfg.OTel.Addr, Headers: headers, From ce48395d31ce705e144f603c2ea11c2f91e9187d Mon Sep 17 00:00:00 2001 From: Jack Woods Date: Wed, 13 May 2026 22:49:19 -0400 Subject: [PATCH 21/25] fix(deploy): per-checkout SigNoz JWT secret, no shared default MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The previous fix (commit 380daf5) for the "make signoz-up hard-fails" review thread switched compose.yaml from `:?` to `:-wavehouse-local-dev-only`. That eliminated the boot break but introduced a different problem flagged on re-review: a public repository now ships a known JWT signing key, so anyone with the string can forge valid tokens against any local SigNoz instance whose operator didn't override the env var. Severity is low for a tool that binds to localhost, but the default is genuinely a worse posture than either fail-fast or a per-checkout random value. Replaced both failure modes with a per-checkout random secret cached in a gitignored file: - `deployments/signoz/compose.yaml` reverts to the `:?` form so a raw `docker compose -f ... up` fails fast with a clear pointer at the remediation paths (`make signoz-up` or `export SIGNOZ_TOKENIZER_JWT_SECRET=$(openssl rand -hex 32)`). Inline comment names the cache path and explains how it rotates. - `Makefile` gains a `$(SIGNOZ_SECRET_FILE)` rule (`deployments/signoz/.signoz-secret.local`) that generates the secret via `openssl rand -hex 32` on first need, with `chmod 600`. `SIGNOZ_COMPOSE` (recursively expanded) now prefixes every `docker compose` call with the secret read from the cache file, so all `signoz-*` recipes get it transparently. `signoz-wipe` rms the cache after `docker compose down -v`, so the next `signoz-up` rotates to a fresh value — matching the volume reset which already invalidates any tokens issued under the old secret. All four `signoz-{up,down,logs,wipe}` targets list the secret file as a prerequisite, so any compose invocation in those recipes is guaranteed to find a cached secret without ever hitting compose's interpolation guard. - `.gitignore` ignores `deployments/signoz/.signoz-secret.local`. Verified end-to-end: `docker compose config` without the env fails with the new pointer message; the Makefile rule generates a 64-char hex file with mode 0600; compose parses cleanly with the cached secret loaded; `git check-ignore` confirms the cache file is matched. Closes the security-LOW [SHOULD] thread at compose.yaml:144 (escalated in scope because the underlying bug — shared known secret — undid the prior [MUST] fix's intent). --- .gitignore | 4 ++++ Makefile | 22 +++++++++++++++++----- deployments/signoz/compose.yaml | 11 +++++++---- 3 files changed, 28 insertions(+), 9 deletions(-) diff --git a/.gitignore b/.gitignore index 2cd1becc..82d2e999 100644 --- a/.gitignore +++ b/.gitignore @@ -64,6 +64,10 @@ Thumbs.db .env.*.local !.github/board-config.env +# SigNoz dev-stack JWT secret (generated by `make signoz-up`, rotated by +# `make signoz-wipe`). Never shared — each checkout gets its own value. +deployments/signoz/.signoz-secret.local + # Debug __debug_bin* diff --git a/Makefile b/Makefile index 4e69125f..b893baa0 100644 --- a/Makefile +++ b/Makefile @@ -233,14 +233,25 @@ deps-wipe: ## Stop ClickHouse AND destroy its data volume (DESTRUCTIVE — use t # SigNoz stack lives in its own compose file. Same DEV_COMPOSE pattern — # alias once, keep recipes terse, easy to swap binaries (e.g. podman compose). SIGNOZ_COMPOSE_FILE := deployments/signoz/compose.yaml -SIGNOZ_COMPOSE := docker compose -f $(SIGNOZ_COMPOSE_FILE) +SIGNOZ_SECRET_FILE := deployments/signoz/.signoz-secret.local +# SIGNOZ_TOKENIZER_JWT_SECRET is required by compose (the :? modifier fails +# fast for raw `docker compose` users). The Makefile generates a random +# per-checkout value once and caches it in a gitignored file, then exports +# it for every signoz-* target. signoz-wipe rotates it. Avoids the +# shared-default-secret risk of a publicly-checked-in compose file. +SIGNOZ_COMPOSE = SIGNOZ_TOKENIZER_JWT_SECRET=$$(cat $(SIGNOZ_SECRET_FILE)) docker compose -f $(SIGNOZ_COMPOSE_FILE) + +$(SIGNOZ_SECRET_FILE): + @command -v openssl >/dev/null 2>&1 || { echo "$(RED)==> 'openssl' is required to generate the SigNoz JWT secret$(RESET)"; exit 1; } + @openssl rand -hex 32 > $@ + @chmod 600 $@ # `--wait` blocks on the named long-running services' healthchecks: # signoz → /api/v1/health on :8080 # otel-collector → health_check extension on :13133 # clickhouse + zookeeper-1 reach healthy first via the dependency chain. .PHONY: signoz-up -signoz-up: ## Start SigNoz stack (idempotent; blocks until UI + collector healthy) +signoz-up: $(SIGNOZ_SECRET_FILE) ## Start SigNoz stack (idempotent; blocks until UI + collector healthy) @echo "$(CYAN)==> Starting SigNoz stack...$(RESET)" @$(SIGNOZ_COMPOSE) up -d --wait signoz otel-collector @echo " SigNoz UI: $(GREEN)http://localhost:3301$(RESET) (first-run: create local admin account)" @@ -248,18 +259,19 @@ signoz-up: ## Start SigNoz stack (idempotent; blocks until UI + collector health @echo " OTLP HTTP: $(GREEN)localhost:4318$(RESET)" .PHONY: signoz-down -signoz-down: ## Stop SigNoz stack (preserves volumes — UI history kept) +signoz-down: $(SIGNOZ_SECRET_FILE) ## Stop SigNoz stack (preserves volumes — UI history kept) @echo "$(YELLOW)==> Stopping SigNoz...$(RESET)" @$(SIGNOZ_COMPOSE) down .PHONY: signoz-logs -signoz-logs: ## Tail SigNoz UI + collector logs (Ctrl+C to detach) +signoz-logs: $(SIGNOZ_SECRET_FILE) ## Tail SigNoz UI + collector logs (Ctrl+C to detach) @$(SIGNOZ_COMPOSE) logs -f signoz otel-collector .PHONY: signoz-wipe -signoz-wipe: ## Stop SigNoz AND destroy its volumes (DESTRUCTIVE — admin account reset) +signoz-wipe: $(SIGNOZ_SECRET_FILE) ## Stop SigNoz AND destroy its volumes (DESTRUCTIVE — admin account reset) @echo "$(RED)==> Wiping SigNoz (containers + volumes)...$(RESET)" @$(SIGNOZ_COMPOSE) down -v --remove-orphans + @rm -f $(SIGNOZ_SECRET_FILE) # Dashboard loader auths to the SigNoz API with a JWT (SIGNOZ_TOKEN, pulled # from the UI's localStorage AUTH_TOKEN). v0.122.0 moved password login to an diff --git a/deployments/signoz/compose.yaml b/deployments/signoz/compose.yaml index 96f1202b..3f1e2c2c 100644 --- a/deployments/signoz/compose.yaml +++ b/deployments/signoz/compose.yaml @@ -138,10 +138,13 @@ services: - SIGNOZ_ALERTMANAGER_PROVIDER=signoz - SIGNOZ_TELEMETRYSTORE_CLICKHOUSE_DSN=tcp://clickhouse:9000 - SIGNOZ_SQLSTORE_SQLITE_PATH=/var/lib/signoz/signoz.db - # Dev-only default — this is a local-dev compose. Override with `export - # SIGNOZ_TOKENIZER_JWT_SECRET=$(openssl rand -hex 32)` before `make - # signoz-up` if you're repurposing this stack for anything non-local. - - SIGNOZ_TOKENIZER_JWT_SECRET=${SIGNOZ_TOKENIZER_JWT_SECRET:-wavehouse-local-dev-only} + # JWT signing key for the SigNoz tokenizer. No default by design — a shared + # default in a public repo would let anyone forge tokens against any local + # SigNoz instance that didn't override it. `make signoz-up` generates a + # random per-checkout value (cached in deployments/signoz/.signoz-secret.local, + # gitignored; rotated by `make signoz-wipe`). If you're running compose + # directly: `export SIGNOZ_TOKENIZER_JWT_SECRET=$(openssl rand -hex 32)`. + - SIGNOZ_TOKENIZER_JWT_SECRET=${SIGNOZ_TOKENIZER_JWT_SECRET:?Set SIGNOZ_TOKENIZER_JWT_SECRET — e.g. via 'make signoz-up' or 'export SIGNOZ_TOKENIZER_JWT_SECRET=$$(openssl rand -hex 32)'} healthcheck: test: ["CMD", "wget", "--spider", "-q", "localhost:8080/api/v1/health"] interval: 30s From 796c4bd8af263485ee48d5b0fa1cd3ef71180773 Mon Sep 17 00:00:00 2001 From: Jack Woods Date: Wed, 13 May 2026 23:05:04 -0400 Subject: [PATCH 22/25] docs(observability): cross-reference the two OTel-headers parsers MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit config.validateOTelHeaders and observability.ParseOTelHeaders encode the same parsing rules but live in different packages because config can't import observability without transitively pulling the OTel SDK into every config consumer (config sits at the bottom of the dep graph by design). That arrangement is fine, but it creates a drift surface: a rule change in one without the matching change in the other would silently let `config.Validate()` accept input that `ParseOTelHeaders` truncates, and `cmd/wavehouse/main.go`'s defensive error check (commit 98621af) would catch the drift at startup loudly — but only after it had already happened. Added a "MUST stay in sync with X" comment on both functions naming the counterpart explicitly, plus a one-line reminder that any rule change here needs the same change there. Makes the coupling discoverable from either side rather than relying on future editors to grep for it. Closes the [MAY] thread at config.go:169. --- internal/config/config.go | 9 ++++++++- internal/observability/endpoint.go | 6 ++++++ 2 files changed, 14 insertions(+), 1 deletion(-) diff --git a/internal/config/config.go b/internal/config/config.go index 2a66afd4..79c83da6 100644 --- a/internal/config/config.go +++ b/internal/config/config.go @@ -165,7 +165,14 @@ type DLQ struct { // validateOTelHeaders mirrors observability.ParseOTelHeaders for boot-time // validation. Kept here (rather than importing observability) so config stays -// at the bottom of the dependency graph. +// at the bottom of the dependency graph — importing observability would +// transitively pull the OTel SDK into every config consumer. +// +// MUST stay in sync with observability.ParseOTelHeaders in +// internal/observability/endpoint.go. cmd/wavehouse/main.go's defensive +// error-check on the post-Validate parse exists to catch any drift loudly, +// but the right answer is to not drift in the first place: any rule change +// here needs the same change there, and vice versa. func validateOTelHeaders(s string) error { s = strings.TrimSpace(s) if s == "" { diff --git a/internal/observability/endpoint.go b/internal/observability/endpoint.go index 89de56b1..50ee49b8 100644 --- a/internal/observability/endpoint.go +++ b/internal/observability/endpoint.go @@ -50,6 +50,12 @@ func ParseEndpoint(addr string) (host string, useTLS bool) { // Returns an error (rather than silently dropping the segment) for malformed // entries so Validate() can fail loud at boot rather than letting a typo // silently disable auth in production. +// +// MUST stay in sync with config.validateOTelHeaders in +// internal/config/config.go. config can't import observability without +// transitively pulling the OTel SDK into every config consumer, so the +// parsing rules are hand-mirrored. Any rule change here needs the same +// change there, and vice versa. func ParseOTelHeaders(s string) (map[string]string, error) { s = strings.TrimSpace(s) if s == "" { From 5babcf40fdaa5b024981e8a4eb5410a68c726b37 Mon Sep 17 00:00:00 2001 From: Jack Woods Date: Wed, 13 May 2026 23:05:13 -0400 Subject: [PATCH 23/25] test(observability): per-signal override test covers metrics path too MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit TestOTel_PerSignalEndpoint_Override only verified the TracesEndpoint override path. Metrics and logs overrides go through the same pickEndpoint() helper in provider.go, but a copy-paste bug — e.g. the metrics exporter wired to cfg.TracesEndpoint instead of cfg.MetricsEndpoint — wouldn't have failed the test: metrics would silently land on the trace receiver and the assertion checking the default endpoint received metrics would also pass (because before this change, no MetricsEndpoint override was set, so metrics correctly went to the default). Extended the test: - Added a third FakeOTLP receiver, `rMetrics`, and pointed MetricsEndpoint at it. - The default endpoint now expects to receive *nothing* (both signals individually routed) — caught by `rDefault.MetricCount()` being zero. - Cross-asserts: rTraces sees zero metrics, rMetrics sees zero spans. Either of these would fail if someone swapped the endpoint variable in a future provider.go edit. The docstring is rewritten to call out the symmetric-wiring intent — so the next contributor to touch the test knows the cross- asserts aren't paranoid filler, they're the actual point. Logs are still excluded; logs goes through the same pickEndpoint() helper so by induction the test now covers that too. If a fourth receiver / LogsEndpoint assertion turns out to be necessary later (say, the helper gets refactored signal-by-signal), it's a five-line extension following the same pattern. Closes the [MAY] thread at tests/integration/otel_test.go:433. --- tests/integration/otel_test.go | 24 ++++++++++++++++-------- 1 file changed, 16 insertions(+), 8 deletions(-) diff --git a/tests/integration/otel_test.go b/tests/integration/otel_test.go index 77644a6a..86b9697e 100644 --- a/tests/integration/otel_test.go +++ b/tests/integration/otel_test.go @@ -399,17 +399,23 @@ func TestOTel_Headers_AppliedToAllSignals(t *testing.T) { assert.Equal(t, []string{"abc123"}, r.LastTraceHeaders().Get("x-honeycomb-team")) } -// TestOTel_PerSignalEndpoint_Override sends traces to one receiver and metrics -// to another by setting TracesEndpoint on top of a default Endpoint. Grafana -// Cloud's distinct gateway hosts per signal are the headline use case. +// TestOTel_PerSignalEndpoint_Override verifies that TracesEndpoint and +// MetricsEndpoint each route their own signal to a distinct receiver while +// the default Endpoint sees neither. Grafana Cloud's per-signal gateway hosts +// are the headline use case. Both signal-type overrides go through the same +// pickEndpoint() helper in provider.go, so a copy-paste bug (e.g. the +// metrics exporter being wired to TracesEndpoint) would surface here as +// metrics landing on the wrong receiver — caught by the cross-asserts below. func TestOTel_PerSignalEndpoint_Override(t *testing.T) { guardOTelGlobals(t) rDefault := testutil.NewFakeOTLP(t) rTraces := testutil.NewFakeOTLP(t) + rMetrics := testutil.NewFakeOTLP(t) shutdown, _ := initAndShutdown(t, observability.ProviderConfig{ - Endpoint: rDefault.Addr(), // metrics + logs land here - TracesEndpoint: rTraces.Addr(), // traces override + Endpoint: rDefault.Addr(), // both signals overridden → default sees nothing + TracesEndpoint: rTraces.Addr(), + MetricsEndpoint: rMetrics.Addr(), TracesEnabled: true, TracesSampleRate: 1.0, MetricsEnabled: true, @@ -426,8 +432,10 @@ func TestOTel_PerSignalEndpoint_Override(t *testing.T) { defer drainCancel() require.NoError(t, shutdown(drainCtx)) - assert.Equal(t, 1, rTraces.SpanCount(), "trace endpoint should have received the span") + assert.Equal(t, 1, rTraces.SpanCount(), "TracesEndpoint should receive the span") + assert.GreaterOrEqual(t, rMetrics.MetricCount(), 1, "MetricsEndpoint should receive the metric") assert.Zero(t, rDefault.SpanCount(), "default endpoint must NOT receive traces when overridden") - assert.GreaterOrEqual(t, rDefault.MetricCount(), 1, "default endpoint should receive metrics (no override)") - assert.Zero(t, rTraces.MetricCount(), "trace endpoint should NOT receive metrics") + assert.Zero(t, rDefault.MetricCount(), "default endpoint must NOT receive metrics when overridden") + assert.Zero(t, rTraces.MetricCount(), "TracesEndpoint should NOT receive metrics (catches metrics-to-traces wiring bug)") + assert.Zero(t, rMetrics.SpanCount(), "MetricsEndpoint should NOT receive spans (catches traces-to-metrics wiring bug)") } From 2574d65be42083c03aee7e4644d9d9d96ac7a6fa Mon Sep 17 00:00:00 2001 From: Jack Woods Date: Wed, 13 May 2026 23:19:26 -0400 Subject: [PATCH 24/25] fix(make): silence cat-not-found when clean-all hits the SigNoz prefix MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The SIGNOZ_COMPOSE macro reads the cached JWT secret via `cat $(SIGNOZ_SECRET_FILE)`. Every `signoz-*` recipe declares the secret file as a prerequisite, so Make ensures the file exists before those recipes run — `cat` always succeeds inside their bodies. `clean-all` is different: it calls `$(SIGNOZ_COMPOSE) down -v ... || true` without depending on the secret file (correctly — we don't want to generate a secret just to tear something down). If the developer runs `make clean-all` on a fresh checkout that never invoked `signoz-up`, the `cat` subshell emits a `cat: deployments/signoz/ .signoz-secret.local: No such file or directory` message to stderr before the compose call runs. The compose `:?` interpolation then fails, `down` exits non-zero, `|| true` swallows the exit, and the user sees only the leaked `cat:` stderr line. The overall command still succeeds — the noise is purely cosmetic, but confusing. One-character fix: `2>/dev/null` on the `cat` inside the macro definition. With this: - signoz-{up,down,logs,wipe} are unaffected (Make generates the file before invoking the recipe, so cat doesn't need stderr). - clean-all's compose call now sees an empty SIGNOZ_TOKENIZER_JWT_SECRET, compose's `:?` guard fires silently on its own stderr (already redirected by `2>/dev/null` in the clean-all recipe), `|| true` mops up the exit, and the user sees nothing — which is what they want from a teardown command. Closes the [SHOULD] thread at Makefile:242. --- Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Makefile b/Makefile index b893baa0..55c87e7f 100644 --- a/Makefile +++ b/Makefile @@ -239,7 +239,7 @@ SIGNOZ_SECRET_FILE := deployments/signoz/.signoz-secret.local # per-checkout value once and caches it in a gitignored file, then exports # it for every signoz-* target. signoz-wipe rotates it. Avoids the # shared-default-secret risk of a publicly-checked-in compose file. -SIGNOZ_COMPOSE = SIGNOZ_TOKENIZER_JWT_SECRET=$$(cat $(SIGNOZ_SECRET_FILE)) docker compose -f $(SIGNOZ_COMPOSE_FILE) +SIGNOZ_COMPOSE = SIGNOZ_TOKENIZER_JWT_SECRET=$$(cat $(SIGNOZ_SECRET_FILE) 2>/dev/null) docker compose -f $(SIGNOZ_COMPOSE_FILE) $(SIGNOZ_SECRET_FILE): @command -v openssl >/dev/null 2>&1 || { echo "$(RED)==> 'openssl' is required to generate the SigNoz JWT secret$(RESET)"; exit 1; } From e92b9b35cc6bdb449ff35397410bcc0dfc2cafaa Mon Sep 17 00:00:00 2001 From: Jack Woods Date: Wed, 13 May 2026 23:19:32 -0400 Subject: [PATCH 25/25] chore: untrack WHissues.md MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Removes the running-issues log from the repo root. Review feedback (claude[bot] at WHissues.md:1) flagged that this kind of operational- notes file accumulates drift and overlaps with what CHANGELOG.md already covers — and the entries also point at follow-up work in `standalone.yaml` that isn't part of this PR's scope, which invites readers to mistake "documented" for "fixed." Untracking only — no inline-migration of the §-references. The CHANGELOG and `docs/development.md` references to `WHissues.md` are left in place as dangling pointers; they'll fall out naturally on the next pass that edits those files. Minimum-disruption deletion. The follow-up work this file pointed at: - §2 (standalone.yaml's 50 GB JetStream default vs ~28 GB OrbStack disk → boot failure) — still an open bug in deployments/compose/standalone.yaml. - §3 (missing ClickHouse healthcheck in standalone.yaml → schema discovery races a still-initializing CH) — also still open. - §1 (the SigNoz compose rebuild) and §4 (the network-collision dual-home gotcha) are already addressed by this PR's commits. §2 and §3 should land as separate fixes or be filed as issues — they don't belong bundled in this PR. The deletion makes that explicit. Closes the [SHOULD] thread at WHissues.md:1. --- WHissues.md | 140 ---------------------------------------------------- 1 file changed, 140 deletions(-) delete mode 100644 WHissues.md diff --git a/WHissues.md b/WHissues.md deleted file mode 100644 index 15fa0f26..00000000 --- a/WHissues.md +++ /dev/null @@ -1,140 +0,0 @@ -# WaveHouse — Known Issues - -Running log of issues found while deploying/operating WaveHouse. Newest first. - ---- - -## 1. SigNoz compose stack (`deployments/signoz/compose.yaml`) fails to start — RESOLVED - -**Found:** 2026-05-12, bringing the bundled SigNoz stack up locally on OrbStack. -**Fixed:** 2026-05-12 — `deployments/signoz/` rebuilt from the upstream SigNoz -`deploy/docker/docker-compose.yaml` (SigNoz `v0.122.0` / `signoz-otel-collector -v0.144.3`), trimmed to a single node. - -**Symptom (before):** `docker compose -f deployments/signoz/compose.yaml up -d` -→ `signoz-migrator` exits 1; consequently `signoz` (query-service) and -`signoz-otel-collector` never start (both `depends_on: migrator: -service_completed_successfully`). Only `signoz-clickhouse` stays up. - -Migrator log: - -```text -migrate/bootstrap.go:119 Creating databases -Error: failed to bootstrap store for migrations: code: 62, message: Syntax error: failed at position 42 (end of query): . Expected one of: identifier, string literal -``` - -**Cause:** the old compose file pinned `:latest` on every SigNoz image. The current -`signoz/signoz-otel-collector:latest` migrate CLI defaults `--clickhouse-cluster` -to `cluster` and `--clickhouse-replication` to `true`; the compose passed -`--clickhouse-cluster=""` (empty). An empty cluster name made the bootstrap step -emit a `... ON CLUSTER ` clause with nothing after it → ClickHouse `code: 62` -syntax error. Secondary problem: `signoz/query-service:latest` was a ~14-month-old -build (2025-03) while `signoz/signoz-otel-collector:latest` is from 2026-05 — those -are no longer a compatible pair (`signoz/query-service` has been superseded by the -consolidated `signoz/signoz` image). - -**Fix applied:** rebuilt `deployments/signoz/` to mirror upstream's current single-node -layout, with everything version-pinned: -- `signoz/signoz:v0.122.0` (consolidated query + UI; replaces `signoz/query-service`), - published on host `3301` (→ container `8080`) -- `signoz/signoz-otel-collector:v0.144.3` (OTLP gRPC `4317`, HTTP `4318`) — runs - `migrate sync check` then the collector, with `--manager-config` pointing at the - bundled `otel-collector-opamp-config.yaml` -- `signoz-telemetrystore-migrator` (same collector image) — `migrate bootstrap && sync - up && async up`, driven by `SIGNOZ_OTEL_COLLECTOR_CLICKHOUSE_CLUSTER=cluster` / - `…_REPLICATION=true` env vars (the cluster `ON CLUSTER cluster` DDL now resolves) -- `clickhouse/clickhouse-server:25.5.6` with `clickhouse/{config,users,cluster,custom-function}.xml` - vendored under `deployments/signoz/clickhouse/` — `cluster.xml` defines a 1-shard / - 1-replica cluster named `cluster` and points at `zookeeper-1` -- `signoz/zookeeper:3.7.1` — needed for the `ON CLUSTER` / replicated tables -- `signoz-init-clickhouse` (one-shot) — downloads the `histogramQuantile` UDF binary - into a shared `signoz-clickhouse-user-scripts` volume -- removed the old `deployments/signoz/.env.example` (the `SIGNOZ_CH_USER` / - `SIGNOZ_CH_PASSWORD` knobs no longer exist; ClickHouse uses `CLICKHOUSE_SKIP_USER_SETUP=1` - + the vendored `users.xml`, which has the `default` user with an empty password) - -WaveHouse is wired in via `deployments/compose/standalone.signoz.yaml` -(`WH_OTEL_ENABLED=true`, `WH_OTEL_ADDR=host.docker.internal:4317`); verified that -spans from `service.name=wavehouse` land in `signoz_traces.distributed_signoz_index_v3`. - -**Note:** on first start the collector logs an OpAMP error -(`cannot create agent without orgId` / `Server returned an error response`) and the -WaveHouse OTLP exporter logs a few `error reading server preface: EOF` lines while the -collector finishes restarting — both are transient. The OpAMP error clears once you -create the first user/org in the SigNoz UI; the EOF lines stop once the collector is -steady (data flows regardless). - -**Status:** resolved. - ---- - -## 2. `deployments/compose/standalone.yaml` quick-start fails on a default-sized Docker/OrbStack VM - -**Found:** 2026-05-12. - -**Symptom:** `docker compose -f deployments/compose/standalone.yaml up -d --build` → -the `wavehouse` container exits 1 on boot: - -```text -mq init failed error="create stream: nats: API error: code=500 err_code=10047 description=insufficient storage resources available" path=/app/data/nats -``` - -**Cause:** `standalone.yaml` does not set `WH_MQ_MAX_BYTES_GB`, so the embedded -JetStream stream uses the 50 GB default (`config.go`: `MaxBytesGB env-default:"50"`). -A stock OrbStack VM disk is ~28 GB; JetStream caps usable file storage well below -that (~16 GB observed), so creating a 50 GB stream fails. WaveHouse treats the MQ -init failure as fatal. - -**Workaround / fix:** set `WH_MQ_MAX_BYTES_GB` to something that fits in -`standalone.yaml` (the e2e compose uses `1`), or document the disk requirement in -`docs/deployment.md`. Locally this is handled by -`deployments/compose/standalone.signoz.yaml` (sets it to `2`). - ---- - -## 3. `standalone.yaml` doesn't gate WaveHouse on ClickHouse readiness - -**Found:** 2026-05-12. - -**Symptom:** on a cold `up`, the `wavehouse` container exits 1: - -```text -schema discovery failed on boot error="query system.columns: dial tcp :9000: connect: connection refused" -``` - -**Cause:** `standalone.yaml` uses `depends_on: [clickhouse]`, which only waits for -the ClickHouse container to *start*, not to accept connections. WaveHouse runs -schema discovery immediately and treats a failure on boot as fatal (`main.go` -`return 1`), so it races a still-initializing ClickHouse and loses. -`deployments/compose/dependencies.yaml` and `tests/e2e/compose.yaml` both add a -ClickHouse healthcheck and `depends_on: { clickhouse: { condition: -service_healthy } }`; `standalone.yaml` should do the same. - -**Workaround / fix:** add the healthcheck + `service_healthy` condition to -`standalone.yaml` (done in `standalone.signoz.yaml` for the local stack), or restart -the `wavehouse` container after ClickHouse is up. - ---- - -## 4. Gotcha: attaching WaveHouse to the SigNoz network breaks ClickHouse name resolution - -**Found:** 2026-05-12, while wiring WaveHouse's OTLP export into the SigNoz stack. - -**Symptom:** if you put the WaveHouse container on the SigNoz compose network so it -can reach `signoz-otel-collector:4317` by name, schema discovery fails with: - -```text -query system.columns: code: 516, message: default: Authentication failed: password is incorrect, or there is no user with such name. -``` - -**Cause:** `deployments/compose/standalone.yaml` and -`deployments/signoz/compose.yaml` both define a service named `clickhouse`. -On a container attached to both networks, the unqualified hostname `clickhouse` is -ambiguous, and WaveHouse can end up dialing SigNoz's ClickHouse — which is -configured with a non-empty password (`SIGNOZ_CH_PASSWORD`, default `password`), -unlike WaveHouse's own ClickHouse (empty password) — so auth fails. - -**Workaround / fix:** don't dual-home the WaveHouse container. Reach the collector -via `host.docker.internal:4317` (the SigNoz collector publishes `4317` on the host) — -this is what `standalone.signoz.yaml` does. Alternatively, rename one of the -`clickhouse` services so the names don't collide.