jfwoods · May 13, 2026 · May 13, 2026 · May 13, 2026 · May 13, 2026 · May 13, 2026
diff --git a/.gitignore b/.gitignore
@@ -64,6 +64,10 @@ Thumbs.db
 .env.*.local
 !.github/board-config.env
 
+# SigNoz dev-stack JWT secret (generated by `make signoz-up`, rotated by
+# `make signoz-wipe`). Never shared — each checkout gets its own value.
+deployments/signoz/.signoz-secret.local
+
 # Debug
 __debug_bin*
 

diff --git a/AGENTS.md b/AGENTS.md
@@ -42,7 +42,7 @@ Twelve internal packages under `internal/`:
 12. **Structured queries**: Type-safe query AST endpoint (`POST /v1/tables/{table}/query`) validated against schema, with permission enforcement, timestamp bucketing for cache optimization, and `DefaultMaxRows` (10,000) limit cap.
 13. **Named query pipes**: Pre-defined SQL templates (inspired by Tinybird) with parameter binding, role restrictions, and caching. Stored in NATS KV with `.sql` file directory bootstrap.
 14. **TypeScript SDK**: `@wavehouse/sdk` — zero-dependency client with typed query builder, real-time SSE, live queries with smart aggregation classification (incrementable/decomposable/poll), and codegen CLI.
-15. **Observability invariants**: Stdout is *always* 100% — the slog logger fans out to stdout AND OTLP, and stdout sampling would silently hide records that scraping pipelines (Promtail/Alloy/Vector → Loki) are paying to store. Sampling knobs apply only to OTLP push. WARN+ERROR records always export at 100% regardless of `logs.sample_rate` — silently dropping errors during incidents would be a worse failure mode than the cost of forwarding them all (this is a *non-configurable* floor; do not expose it). gRPC OTel exporters dial lazily, so an unreachable collector never blocks startup; transient failures surface via the OTel SDK's error handler. The OTel Prometheus exporter (when enabled) uses a *private* `prometheus.Registry` to avoid leaking process/Go collectors that `prometheus.DefaultRegisterer` auto-registers into our `/metrics` output. When changing the logger, the sampler, or the provider wiring, preserve these invariants.
+15. **Observability invariants**: Stdout is *always* 100% — the slog logger fans out to stdout AND OTLP, and stdout sampling would silently hide records that scraping pipelines (Promtail/Alloy/Vector → Loki) are paying to store. Sampling knobs apply only to OTLP push. WARN+ERROR records always export at 100% regardless of `logs.sample_rate` — silently dropping errors during incidents would be a worse failure mode than the cost of forwarding them all (this is a *non-configurable* floor; do not expose it). gRPC OTel exporters dial lazily, so an unreachable collector never blocks startup; transient failures surface via the OTel SDK's error handler. The OTel Prometheus exporter (when enabled) uses a *private* `prometheus.Registry` to avoid leaking process/Go collectors that `prometheus.DefaultRegisterer` auto-registers into our `/metrics` output. TLS is selected via *scheme sniffing* on `otel.addr` (`https://` → TLS, anything else → plaintext) — there is no separate `otel.tls.enabled` boolean. `otel.headers` is applied uniformly to all OTLP exporters (no per-signal header override); per-signal endpoint overrides (`otel.{traces,metrics,logs}.addr`) are the supported way to split signals across gateways. When changing the logger, the sampler, or the provider wiring, preserve these invariants.
 16. **Bearer-token-only CORS posture**: WaveHouse is a Bearer-token API — `Authorization: Bearer <jwt>` on every authenticated request, no cookies, no session middleware. The CORS middleware (`internal/api/router.go` `corsMiddleware`) deliberately **never** emits `Access-Control-Allow-Credentials`, because (a) we don't need it (Bearer tokens are explicit request headers, not browser-managed credentials) and (b) the historical pairing of `Allow-Credentials: true` with `Allow-Origin: *` is a CORS spec violation that browsers reject. The `cors_allowed_origins` allowlist controls *which origins can read responses*, not cookie scope. CSRF protection is structural: cross-site requests can't smuggle a Bearer token because the browser won't auto-attach `Authorization` headers cross-origin. Do not reintroduce cookie-based auth or `Access-Control-Allow-Credentials` without a separate design discussion — the current posture is the answer to GitHub issues #29 and #30.
 
 ## Code Conventions

diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -17,6 +17,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - **Dependabot auto-merge no longer slipped past CI on consolidated-pipeline PRs** (`main branch protection` ruleset; `.github/workflows/project-orchestrator.yml`; `.github/workflows/claude-review.yml`; `AGENTS.md`): on commit `93b3206` the previously six-job CI was collapsed into a single job named `CI`, but three places still referenced the old multi-job names — the ruleset's `required_status_checks` (`Validate` + `Admin approval` only — `CI` not listed), the `bot-clean` check in `project-orchestrator.yml` (looking for `Build`/`Validate`/`Lint`/`Test`/`Integration Tests`/`SDK Tests`), and `REQUIRED_CHECKS` in `claude-review.yml` (same six). Net effect: a Dependabot PR opened at T+0 would have its `Validate` and `Admin approval` checks satisfied within ~30s (PR-title workflow + the auto-approval workflow), `gh pr merge --auto` would fire, GitHub would see all *required* checks green, and the PR would squash-merge before `ci.yml` even started running. Pre-consolidation runs (PR #90 etc.) didn't expose this because GitHub's auto-merge waits for in-flight checks as a courtesy, and the multi-job CI was producing checks before auto-merge fired — but with one collapsed `CI` check that didn't start until later, the courtesy wait disappeared. Fix: added `CI` to the ruleset's required-checks (now `Admin approval` / `CI` / `Validate`) via the `gh api PUT /repos/Wave-RF/WaveHouse/rulesets/15353356` round-trip, and updated both bot-clean lists to look for `CI` + `Validate`. AGENTS.md note about the required-check set updated to match. Diagnosis steps recorded inline as code comments in both workflow files so future-me knows where to look if the check name changes again.
 
 ### Added
+- **Local SigNoz dev stack + Makefile integration + version-controlled dashboards** (`deployments/signoz/compose.yaml`, `deployments/signoz/clickhouse/{config,users,cluster,custom-function}.xml`, `deployments/signoz/otel-collector-opamp-config.yaml`, `deployments/signoz/dashboards/{wavehouse-overview,wavehouse-runtime-internals}.json`, `deployments/signoz/dashboards/README.md`, `deployments/signoz/load-dashboards.sh`, `deployments/compose/standalone.signoz.yaml`, `Makefile`, `WHissues.md`, `docs/src/content/docs/{deployment,development}.md`, `README.md`): closes [#121](https://github.com/Wave-RF/WaveHouse/issues/121). Replaces the hand-rolled `deployments/signoz/docker-compose.yaml` (pinned `:latest` everywhere and broke against the current `signoz-otel-collector` migrate CLI — see `WHissues.md` §1) with a version-pinned single-node mirror of upstream SigNoz `v0.122.0`'s `deploy/docker/docker-compose.yaml`: `signoz/signoz:v0.122.0` (consolidated UI/API on `:3301`) + `signoz/signoz-otel-collector:v0.144.3` (OTLP gRPC `:4317` / HTTP `:4318`) + dedicated `signoz-telemetrystore-migrator` (gates `signoz` and `otel-collector` on `service_completed_successfully`) + `clickhouse-server:25.5.6` + `zookeeper:3.7.1` + one-shot `init-clickhouse` (fetches the `histogramQuantile` UDF). Healthchecks added on the collector via `/dev/tcp` + `:13133/health_check` so `--wait` is meaningful. Headers comment now points operators at `make signoz-up`. Compose `container_name:` dropped throughout so the project-prefix naming works and multiple instances don't collide. New `##@ Observability` Make targets — `signoz-up` (idempotent, `--wait` on UI + collector), `signoz-down` (preserves admin account), `signoz-logs`, `signoz-wipe` (volumes + admin reset), `signoz-dashboards` (upserts via `load-dashboards.sh`; fail-fast on missing `SIGNOZ_TOKEN`), and `dev-obs` (`deps-up` + `signoz-up` + `air` with `WH_OTEL_ENABLED=true WH_OTEL_ADDR=127.0.0.1:4317`; auto-loads dashboards when `SIGNOZ_TOKEN` is in env, otherwise prints a pointer). `clean-all` now tears down the SigNoz project too. Two dashboards ship as version-controlled JSON — `wavehouse-overview` (HTTP traffic / latency / OTLP intake; widgets scoped to `service.name=wavehouse` so they don't aggregate other services that may also report to a local SigNoz) and `wavehouse-runtime-internals` (Go runtime, embedded NATS, ingest pipeline, payload sizes). `load-dashboards.sh` upserts by title against the running stack; auth is **token-only** because SigNoz `v0.122.0` moved password login to `/api/v2/sessions/email_password` and requires an org UUID that isn't externally discoverable, so the legacy `SIGNOZ_EMAIL`+`SIGNOZ_PASSWORD` branch dead-ended silently — the JWT the SPA keeps in `localStorage` under `AUTH_TOKEN` is one DevTools click away and is what `SIGNOZ_TOKEN` now wants. `standalone.signoz.yaml` wires the containerized `standalone.yaml` WaveHouse into the SigNoz stack via `host.docker.internal:4317`, caps `WH_MQ_MAX_BYTES_GB` for small dev disks (`WHissues.md` §2), and gates WaveHouse on a ClickHouse healthcheck (§3). Don't dual-home a WaveHouse container onto both SigNoz and `standalone.yaml` ClickHouse networks (§4 — name resolution collision; the `host.docker.internal` route sidesteps this).
+- **Direct-to-cloud OTLP — TLS + auth headers, no sidecar required** (`internal/config/config.go`, `internal/observability/provider.go`, `internal/observability/endpoint.go`, `internal/testutil/otlp.go`, `cmd/wavehouse/main.go`, `tests/integration/otel_test.go`, `docs/src/content/docs/{configuration,deployment}.md`, `AGENTS.md`): closes [#97](https://github.com/Wave-RF/WaveHouse/issues/97). `otel.addr` is now scheme-aware — `https://` selects TLS (system root CAs by default), `http://` or a bare `host:port` stays plaintext gRPC (backward-compat with the prior `WithInsecure()` default). New `otel.headers` (`WH_OTEL_HEADERS`) takes the OTel spec env-var format — comma-separated `key=value`, values may contain `=` so base64 padding round-trips — and applies the parsed map as gRPC metadata to every OTLP exporter (traces, metrics, logs). Together these unlock direct push to Honeycomb and Grafana Cloud's OTLP gateway without a sidecar. Datadog does not publish a direct-to-cloud OTLP endpoint — its supported path remains the DDOT Collector embedded in the Datadog Agent (host or k8s), which WaveHouse reaches as a plaintext local OTLP receiver on `127.0.0.1:4317` (no headers — the API-key auth lives on the Agent); see `docs/deployment.md` for the worked example. Per-signal endpoint overrides (`otel.{traces,metrics,logs}.addr`) point one signal at a different gateway than the top-level default (Grafana Cloud uses distinct trace/metric/log hosts in some regions); empty means inherit. Headers always apply uniformly — there is intentionally no per-signal header override. Test surface: `FakeOTLP` gains a `NewFakeOTLPTLS(t)` constructor (ephemeral self-signed cert + matching client `*tls.Config`) and per-signal `LastTraceHeaders / LastMetricHeaders / LastLogHeaders` accessors via captured gRPC metadata. New integration tests pin the TLS dial path, headers-on-all-signals propagation, and per-signal endpoint splitting. Validation rejects malformed `otel.headers` (missing `=`, empty key) at boot rather than letting a typo silently disable auth in production. mTLS / client-certificate auth is not yet supported.
 - **Astro / Starlight documentation site at `wavehouse.dev`** (`docs/`, `Makefile`, `.github/dependabot.yml`, `.github/workflows/ci.yml`, `.gitignore`, `.vscode/launch.json`): full content + tooling for the public docs (Getting Started, Why WaveHouse, Architecture, API Reference, TypeScript SDK, Configuration, Deployment, Development). Build-time mermaid via `rehype-mermaid` (inline-svg strategy, no client-side JS), LaTeX via `remark-math` + `rehype-katex`, image zoom via `starlight-image-zoom`, expressiveCode with github dark/light + `env`/`dns` Shiki language aliases. PostHog frontend telemetry (key is public by design — embedded in every visitor's browser). `editLink` pointed at `main`; `lastUpdated` on. Sidebar lives in `docs/src/config/sidebar.ts` as the single source of truth, consumed by both Starlight rendering and the LLM-friendly outputs below. Cloudflare Workers + Static Assets deployment via `docs/wrangler.jsonc` (no auto-deploy yet — wired up when the GH Actions workflow lands separately). Make targets stay minimal: `dev-docs` (Astro dev server on :4321), `build-docs`, `preview-docs` — the last serves the production build through `wrangler dev`, so the Cloudflare Worker's content negotiation (`Accept: text/markdown` / LLM-bot User-Agent → `.md` twin) is exercised end-to-end the same way it will be in production.
 - **Per-page `.md` twin + `llms.txt` family via `starlight-llm-tools`** (`docs/astro.config.mjs`, `docs/worker/index.ts`, `docs/wrangler.jsonc`): every doc is also served as raw markdown at `<path>.md` with a navigation header (Section / Subpages / Related / HTML version pointers), plus three concatenated views — `/llms.txt` manifest, `/llms-full.txt` (every page, sidebar-ordered), `/llms-small.txt` (overview pages only). Page-title slot gains a **Copy Markdown** button and **Open with AI** dropdown (Claude / ChatGPT / Cursor). Content negotiation is handled by a Cloudflare Worker that re-exports the `cloudflare-md-router` package's default handler in one line — `Accept: text/markdown` or known LLM-bot User-Agent (GPTBot, ClaudeBot, PerplexityBot, Applebot-Extended, etc.) gets the `.md` twin transparently, with a fall-through to the HTML response when the twin doesn't exist.
 - **Two extracted plugin packages, MIT-licensed**: [`github.com/Wave-RF/cloudflare-md-router`](https://github.com/Wave-RF/cloudflare-md-router) (Workers handler + `createMdRouter()` factory + extensible bot-UA regex; consumed in `docs/worker/index.ts` as a one-line re-export) and [`github.com/Wave-RF/starlight-llm-tools`](https://github.com/Wave-RF/starlight-llm-tools) (Starlight plugin that auto-injects the four routes, the two components, and the `PageTitle` override; optionally calls into `starlight-glossary/transform` when present). Both pinned via `github:Wave-RF/...` in `docs/package.json`; lockfile pins the resolved commit SHAs. Pre-npm-publish state — switch to normal version specifiers when the packages land on npm.

diff --git a/Makefile b/Makefile
@@ -228,6 +228,85 @@ deps-wipe: ## Stop ClickHouse AND destroy its data volume (DESTRUCTIVE — use t
 	@echo "$(RED)==> Wiping ClickHouse (containers + volumes)...$(RESET)"
 	@$(DEV_COMPOSE) down -v --remove-orphans
 
+##@ Observability
+
+# SigNoz stack lives in its own compose file. Same DEV_COMPOSE pattern —
+# alias once, keep recipes terse, easy to swap binaries (e.g. podman compose).
+SIGNOZ_COMPOSE_FILE := deployments/signoz/compose.yaml
+SIGNOZ_SECRET_FILE  := deployments/signoz/.signoz-secret.local
+# SIGNOZ_TOKENIZER_JWT_SECRET is required by compose (the :? modifier fails
+# fast for raw `docker compose` users). The Makefile generates a random
+# per-checkout value once and caches it in a gitignored file, then exports
+# it for every signoz-* target. signoz-wipe rotates it. Avoids the
+# shared-default-secret risk of a publicly-checked-in compose file.
+SIGNOZ_COMPOSE       = SIGNOZ_TOKENIZER_JWT_SECRET=$$(cat $(SIGNOZ_SECRET_FILE)) docker compose -f $(SIGNOZ_COMPOSE_FILE)
+
+$(SIGNOZ_SECRET_FILE):
+	@command -v openssl >/dev/null 2>&1 || { echo "$(RED)==> 'openssl' is required to generate the SigNoz JWT secret$(RESET)"; exit 1; }
+	@openssl rand -hex 32 > $@
+	@chmod 600 $@
+
+# `--wait` blocks on the named long-running services' healthchecks:
+#   signoz          → /api/v1/health on :8080
+#   otel-collector  → health_check extension on :13133
+# clickhouse + zookeeper-1 reach healthy first via the dependency chain.
+.PHONY: signoz-up
+signoz-up: $(SIGNOZ_SECRET_FILE) ## Start SigNoz stack (idempotent; blocks until UI + collector healthy)
+	@echo "$(CYAN)==> Starting SigNoz stack...$(RESET)"
+	@$(SIGNOZ_COMPOSE) up -d --wait signoz otel-collector
+	@echo "    SigNoz UI:  $(GREEN)http://localhost:3301$(RESET)  (first-run: create local admin account)"
+	@echo "    OTLP gRPC:  $(GREEN)localhost:4317$(RESET)"
+	@echo "    OTLP HTTP:  $(GREEN)localhost:4318$(RESET)"
+
+.PHONY: signoz-down
+signoz-down: $(SIGNOZ_SECRET_FILE) ## Stop SigNoz stack (preserves volumes — UI history kept)
+	@echo "$(YELLOW)==> Stopping SigNoz...$(RESET)"
+	@$(SIGNOZ_COMPOSE) down
+
+.PHONY: signoz-logs
+signoz-logs: $(SIGNOZ_SECRET_FILE) ## Tail SigNoz UI + collector logs (Ctrl+C to detach)
+	@$(SIGNOZ_COMPOSE) logs -f signoz otel-collector
+
+.PHONY: signoz-wipe
+signoz-wipe: $(SIGNOZ_SECRET_FILE) ## Stop SigNoz AND destroy its volumes (DESTRUCTIVE — admin account reset)
+	@echo "$(RED)==> Wiping SigNoz (containers + volumes)...$(RESET)"
+	@$(SIGNOZ_COMPOSE) down -v --remove-orphans
+	@rm -f $(SIGNOZ_SECRET_FILE)
+
+# Dashboard loader auths to the SigNoz API with a JWT (SIGNOZ_TOKEN, pulled
+# from the UI's localStorage AUTH_TOKEN). v0.122.0 moved password login to an
+# endpoint that needs an org UUID we can't externally discover, so the token
+# is the only supported path. The guard fails fast with a pointer to the
+# first-run flow rather than letting the loader emit a less obvious 401.
+.PHONY: signoz-dashboards
+signoz-dashboards: ## Upsert WaveHouse dashboards into local SigNoz (needs SIGNOZ_TOKEN)
+	@if [ -z "$$SIGNOZ_TOKEN" ]; then \
+		echo "$(RED)==> Set SIGNOZ_TOKEN.$(RESET)"; \
+		echo "    First-run: open http://localhost:3301, create your admin account,"; \
+		echo "    then copy AUTH_TOKEN from DevTools → Application → Local Storage."; \
+		exit 1; \
+	fi
+	@deployments/signoz/load-dashboards.sh
+
+# dev-obs = `make dev` + the SigNoz layer + WaveHouse pointed at the
+# collector. `WH_OTEL_ENABLED` is the only flip needed; `WH_OTEL_ADDR`
+# already defaults to 127.0.0.1:4317 in internal/config/config.go but is
+# set explicitly so the recipe doubles as documentation.
+.PHONY: dev-obs
+dev-obs: deps-up signoz-up $(AIR) ## Hot-reload dev server with SigNoz observability
+	@echo "$(CYAN)==> Starting WaveHouse with SigNoz observability$(RESET)"
+	@echo "    WaveHouse:  $(GREEN)http://localhost:8080$(RESET)"
+	@echo "    SigNoz UI:  $(GREEN)http://localhost:3301$(RESET)  (first-run: create local admin account)"
+	@if [ -n "$$SIGNOZ_TOKEN" ]; then \
+		echo "$(CYAN)==> Loading dashboards (SIGNOZ_TOKEN detected)...$(RESET)"; \
+		deployments/signoz/load-dashboards.sh || echo "$(YELLOW)==> Dashboard load failed; continuing$(RESET)"; \
+	else \
+		echo "    Dashboards: run $(CYAN)make signoz-dashboards$(RESET) after creating your SigNoz account"; \
+	fi
+	@WH_OTEL_ENABLED=true \
+	 WH_OTEL_ADDR=127.0.0.1:4317 \
+	 $(AIR) -c .air.toml
+
 ##@ Code Quality
 
 .PHONY: fmt
@@ -571,6 +650,7 @@ clean-all: clean clean-test clean-tools ## Full reset — clean + clean-test + c
 	@echo "$(YELLOW)==> Full reset (dev state, docker)...$(RESET)"
 	@rm -rf data/
 	@$(DEV_COMPOSE) down -v --remove-orphans 2>/dev/null || true
+	@$(SIGNOZ_COMPOSE) down -v --remove-orphans 2>/dev/null || true
 	@docker compose -f tests/e2e/compose.yaml down -v --remove-orphans 2>/dev/null || true
 
 ##@ Tooling

diff --git a/README.md b/README.md
@@ -134,6 +134,8 @@ make dev
 
 WaveHouse will automatically recompile and restart whenever you save a `.go` file.
 
+**With observability:** `make dev-obs` is the same as `make dev`, plus a local [SigNoz](https://signoz.io/) stack on `http://localhost:3301` with WaveHouse already pointed at its OTLP collector. See [docs/development.md § Running with observability](docs/src/content/docs/development.md#running-with-observability--make-dev-obs).
+
 ## 🤝 Contributing
 
 We welcome issues, pull requests, and feedback! Please see our [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines on how to structure your code and run the integration test suites.