diff --git a/AGENTS.md b/AGENTS.md index 98fbe517..92bc788f 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -334,7 +334,7 @@ The TypeScript SDK (`@wavehouse/sdk` in `clients/ts/`) is the canonical client a | Backend change | SDK considerations | | -------------- | ------------------ | -| New user-facing API endpoint | Add a typed client method (in `clients/ts/src/client.ts` or the relevant subsystem file: `query-builder.ts`, `pipes.ts`, `policy.ts`, `stream/`, etc.); update `docs/src/content/docs/sdk.md` | +| New user-facing API endpoint | Add a typed client method (in `clients/ts/src/client.ts` or the relevant subsystem file: `query-builder.ts`, `pipes.ts`, `policy.ts`, `stream/`, etc.); update the matching SDK doc page under `docs/src/content/docs/sdk/` (`queries`, `streaming`, `pipes`, `admin`, or `reference` by topic — plus the API tree in `reference.md`) | | Change to JWT auth / role extraction | Update auth handling in `clients/ts/src/http.ts` and types in `clients/ts/src/client.ts` | | Change to `EventMessage` / ingest event format | Update payload types in `clients/ts/src/` (some are codegen-regenerated — re-run the SDK codegen CLI) | | New / changed structured query AST | Update `clients/ts/src/query-builder.ts` types + builder methods | diff --git a/CHANGELOG.md b/CHANGELOG.md index 25cd275b..da79dccf 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -20,6 +20,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Fixed +- **Docs mobile rendering overhaul — tables, diagrams, chrome, and wrapped headings no longer break at phone widths** (`docs/src/styles/global.css`, `docs/src/components/{ScrollHints (new),MermaidZoom,Footer,ReadingProgress}.astro`): a 15-agent visual audit of every page × 320–1440px × both themes (driven by the PR author's report that "many pages look broken on mobile") found four systemic defects, all shipped-broken before this branch and some made worse by it. (1) **Tables crushed instead of scrolling**: the fixed-layout + `overflow-wrap: anywhere` treatment kept dense tables inside the column by shredding content — at 390px reference tables wrapped identifiers a few characters at a time (`WH_P/ROME/THEU/S_PO/RT`), and even at 1440px fixed equal columns split tokens mid-word. Tables now scroll instead of crushing: a JS wrapper (`ScrollHints.astro`) gives each table a framed scroll container with true auto layout (prose wraps first, scroll only when unbreakable columns can't fit), with a GitHub-style `display:block` scroll fallback when JS is off. (2) **Mermaid diagrams scaled to thumbnails on phones**: standalone diagrams shrank to fit the column (≈3–7px labels — unreadable); they're now wrapped in the same scroller and keep a readable size with horizontal pan (no-JS keeps scale-to-fit), the zoom lightbox uses `safe` centering so a natural-size diagram's left/top edge is reachable on phones (plain centering made overflow start-side unscrollable), and coarse-pointer devices get a visible ⤢ zoom affordance (cursor: zoom-in means nothing on touch). (3) **Translucent fixed chrome double-exposed on mobile**: the glass header and Starlight's sticky "On this page" bar ghosted scrolled content through themselves; both are opaque below 50rem (glass stays on desktop), and the hand-written `-webkit-backdrop-filter` twin was removed because it made LightningCSS treat prefixing as author-managed and drop the standard `backdrop-filter` from the build entirely (Firefox had no blur). (4) **Wrapped headings drew their section rule through the text**: Starlight renders content headings `display: inline` inside `.sl-heading-wrapper` (for the anchor link), so backgrounds/borders fragment per line — the h2 rule struck through every wrapped line on mobile (the pre-branch `border-bottom` had the same bug) and heading block margins silently no-oped; rhythm + rule now live on the wrapper. Plus: every horizontal scroller (code blocks, tables, diagrams, the zoom lightbox's pannable stage) gets a right-edge fade scrim while more content is hidden (cut-off code read as truncated, not scrollable), code ligatures are disabled site-wide (`!=` rendered as `≠` in tables documenting exact operator syntax), the footer keeps three columns at tablet widths, and the reading-progress bar regains visibility (0.9 opacity). A second 15-agent verification round over a fresh corpus then confirmed the fixes and caught the stragglers: scrolling tables gained per-column minimum widths (without floors, auto layout squeezed off-screen prose columns to ~100px whose 10–20 wrapped lines left near-blank viewport-tall rows — later scoped to tables measured as genuinely overflowing, see the Layout bullet below), table code chips no longer break at hyphens (`--url` read as two flags; later refined to wrap between tokens, same bullet), diagrams pin to their Mermaid-stamped natural width inside the pan scroller (a 540px floor still let the widest shrink ~33% to ~5px labels; later scoped to phone widths), aside titles flow as block text so a title's code chip can't detach from its sentence (Starlight's flex title made the chip its own flex item), and the ingest-pipeline state-machine's edge labels were shortened to stop dagre overlapping them at every width. + +- **sdk.md's Live Queries example no longer chains the query builder off `from()` directly** (`docs/src/content/docs/sdk.md`, `docs/src/content/docs/index.mdx`): `wh.from(table)` returns a `TableRef`, which has no `.where()`/`.orderBy()`/`.limit()`/`.liveQuery()` — those live on `QueryBuilder`, entered only via `.select(...)`/`.selectAll()`. The documented example failed to type-check and, copied as-is, threw `TypeError: wh.from(...).where is not a function`; it now starts the chain with `.selectAll()`, and the landing-page SDK showcase (added on this branch, which had faithfully inherited the broken pattern from this example) gets the same fix. Surfaced by the branch's pre-push review; no e2e test exercises that chain shape, which is why it had gone uncaught. + - **Docs search and the mobile menu survive in-page navigation again — restored via the supported `astro-vtbot` Starlight integration, which keeps the SPA view transitions rather than dropping them** (`docs/package.json`, `pnpm-lock.yaml`, `docs/src/components/Head.astro`): the design-token rebuild (#142) added a bare `` (Astro View Transitions) to `Head.astro`, but Starlight does not support a bare router — its lead maintainer warns it breaks search and the mobile menu ([withastro/starlight#2823](https://github.com/withastro/starlight/discussions/2823)). The SPA router swapped the whole header on every navigation, so (a) the previous page's `window` ⌘K handler pointed at a now-detached `` and the next ⌘K threw `InvalidStateError: Failed to execute 'showModal' … The element is not in a Document`, and (b) Pagefind's one-shot `DOMContentLoaded` init never re-ran, so the search box reopened empty; the mobile menu broke by the same header-swap. Rather than give up the SPA model (instant navigation, preserved sidebar scroll) for browser-native cross-document transitions, `Head.astro` now wraps Starlight's `Head` in `astro-vtbot`'s `components/starlight/Base.astro` — the integration #2823 names as the supported path. `Base` supplies the `` plus a `` that swaps only the main content frame and **preserves the rest of the header**, so the search `` and its ⌘K listener are never detached, and a `` that re-runs Starlight's own per-navigation init — it closes the mobile menu, updates the current-page marker, and keeps the existing sidebar DOM when its links are unchanged, preserving scroll position. The docs components keep the `astro:page-load`/`astro:after-swap` re-init they have on `main` (navigations are still Astro View Transitions), so nothing else changes. Verified on a production build (`astro build` + the Cloudflare Worker) with Playwright: SPA navigation preserves the JS context (no full reload); ⌘K search opens and returns results after navigation with no `showModal` error and exactly one `` in the document; and the mobile menu opens, auto-closes on navigation, then re-opens afterward (the original breakage). - **Mermaid node labels no longer clip their last glyph, and the comparison diagram pair stacks vertically instead of shrinking side-by-side** (`docs/src/config/mermaid-theme.mjs`, `docs/package.json`, `pnpm-lock.yaml`, `docs/src/styles/global.css`, upstream [`astro-themed-mermaid`](https://github.com/Wave-RF/astro-themed-mermaid) → v0.3.0): the build-time Chromium measured each node's box at the default font-weight 400 while `global.css` displays labels at weight 500, so every box came out ~1px too narrow and the longest label clipped on the right (`Buffer Consumer` → `Buffer Consume` — present across all 116 nodes on the live preview). Fixed upstream via a new `measurementCss` option on `astro-themed-mermaid` that injects label metrics into the render page so Mermaid measures what the browser shows; `mermaid-theme.mjs` passes the node-label weight + letter-spacing (plus a 1.5px safety pad) and resets cluster titles to their default measurement (they render as `overflow:visible` pills sized separately by the plugin, so a weight bump there would only mis-center them). Selectors are bare (`.nodeLabel p`, not `svg[aria-roledescription…] .nodeLabel p`) because Mermaid measures the label before the flowchart `` wrapper exists. Separately, the `.diagram-pair` wrapper (the DIY-vs-WaveHouse comparison on `why-wavehouse`) dropped its ≥1500px flex-row variant: two detailed diagrams side-by-side each shrank to ~450px on a wide monitor, so they now always stack vertically at the full content width — matching the new AGENTS.md "Authoring Mermaid diagrams" house style (favour vertical/`TB` over `LR`). - **Embedded NATS no longer installs its own process-wide SIGINT handler** (`internal/mq/embedded.go`, `internal/testutil/natsjs.go`): closes #287. nats-server's `Start()` registers a signal handler unless `NoSigs` is set, and its SIGINT path is `Shutdown(); WaitForShutdown(); os.Exit(0)` — racing WaveHouse's own graceful shutdown on every Ctrl-C/SIGTERM-adjacent stop. Depending on who won: the NATS handler's `os.Exit(0)` skipped all of `run()`'s deferred cleanup (dedup/cache/ClickHouse close) while still exiting 0, or the two concurrent `Server.Shutdown()` calls panicked (`close of nil channel` in `shutdownEventing`, nats-server v2.14.0 events.go:1907) with exit status 2 — and a panic exit also skips Go coverage flushing, which surfaced as the e2e suite suddenly reporting 0.0% coverage in `make ci` (the orchestrator SIGINTs the instrumented binary precisely to flush counters). Now `NoSigs: true`: WaveHouse's shutdown path is the single lifecycle owner and already stops the embedded server via `EmbeddedNATS.Close()`; the twin options literal in `testutil.NewJetStream` (pipes/policy KV tests) gets the same flag so a Ctrl-C during a local test run can't `os.Exit(0)` the test binary mid-suite. Removing the early-exit also exposed how long a *clean* graceful exit actually takes when no OTel collector is listening — ~15s of serialized exporter dial-backoff (filed as #288) — so the e2e orchestrator's SIGINT-to-kill budget rises 10s → 30s (`scripts/orchestrator/main.go`; fast exits are unaffected, the wait selects on process exit). Both shutdown paths now also `WaitForShutdown()` after `Shutdown()` — owning the lifecycle means waiting it out, so the process can't exit (or `t.TempDir()` cleanup race) while JetStream is still tearing down. @@ -35,18 +39,19 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Changed +- **Docs site polish: navigation + reading progress bars, branded search/asides/404, and client-side error tracking** (`docs/src/components/{Head,Footer,PostHog,ProgressBar (new),ReadingProgress (new)}.astro`, `docs/src/content/docs/404.md`, `docs/src/styles/global.css`): builds on the astro-vtbot SPA integration above. Slow navigations get a thin accent progress bar — a new self-hosted `ProgressBar.astro` on vtbot's `loading()` hook, because vtbot's own `` imports Swup's plugin from unpkg.com at runtime in every visitor's browser (a third-party CDN script in prod, against the repo's supply-chain bar; verified present in the built bundle before replacing). Doc pages additionally get a reading-progress bar (`ReadingProgress.astro`, rendered from the sidebar footer branch): a 2px accent line under the header filling with the scrolled-past fraction of the article body — the "how far through this page am I" signal the right-rail TOC gives desktop but mobile had nothing for; it spans only the content column (offset past the sidebar once that exists, hidden entirely ≥72rem where the TOC takes over), ignores footer chrome, hides on articles shorter than the viewport and under the open mobile menu, and updates via one rAF-throttled passive listener that survives view transitions. (A direction-aware page slide + title morph were also tried here and ripped out on review — too much motion for docs; a reduced-motion media query zeroing all view-transition animations stays as a belt-and-braces guard.) The search dialog — previously stock Pagefind chrome — now sits on the design system (surface/border/radius frame with a gentle reduced-motion-aware pop, code-surface input with accent focus, card result groups with a header band + surface-tint row hover replacing Starlight's bare bright outline, accent `` hits, branded clear/load-more controls, blurred backdrop), as do asides (the stock blue/purple/orange/red tokens were never remapped, so the first `:::note` authored would have landed off-palette), prose `kbd`/`mark`, heading-anchor hover, inner-surface scrollbars, and the mobile menu button (stock inverted to a white circle in dark mode). With asides on the design system, nineteen titled note-blockquotes across api/pipes/access-control/sdk/development became real `:::note`/`:::caution` asides (titles promoted to aside labels; the bodies are unchanged — short untitled clarification quotes stay blockquotes), and the sidebar gained a "Reference" group (API Reference + TypeScript SDK out of "Guides" — lookup material, not narrative; no URLs changed). Getting Started gains a "Troubleshooting first runs" section — the five first-session trip-ups (schema-refresh 404, pre-flush empty query, fail-closed 403 on non-demo tables, port conflicts, cold-start timing), each already documented behavior, now answered where a new user actually hits them — and its SDK pointer became a `:::tip` aside. A fresh-eyes screenshot audit then tightened the loose ends: Starlight's stock tab underline (a `box-shadow` that survived the segmented restyle as floating arcs under inactive tabs), a bottom scrim where search results hard-clipped against the dialog's rounded edge, an 88% glass header (scrolled body text bled through at 78%), the mobile menu button joining the search trigger's bordered-surface control family in both states, and full-width stacked hero CTAs on small phones. The landing page gains an SDK showcase — Ingest / Query / Live-updates tabs of `@wavehouse/sdk` snippets (verified against sdk.md's documented API) under a "Query it like a database. Subscribe to it like a socket." section — with Starlight's `` restyled as a segmented control, and the lone "How it ships" card folded into the quickstart prose. The 404 page replaces its bare title + bullet list with a "signal lost" treatment: the landing's grid + glow atmosphere, a mono 404, an EKG-style trace (heartbeat decaying into a flatline, drawn by a looping monitor-sweep dash animation over a live→dead gradient, with a blinking sample dot; static full trace for reduced motion), and the card-style quick links — its custom `h1` replaces the hidden stock title panel so the page keeps exactly one heading. PostHog gains `capture_exceptions: true`, surfacing broken docs JS (LiveDemo SSE, copy buttons, theme scripts) in error tracking; the SPA pageview capture (`capture_pageview: 'history_change'`) was verified against vtbot's lifecycle docs and is unchanged. - **Docs "Last updated" footer shows each page's real last-edit date, and robots.txt declares open AI content signals** (`.github/workflows/ci.yml`, `docs/public/robots.txt`): Starlight's `lastUpdated` reads git history, but CI checked out with the default shallow (depth-1) clone, so every page silently fell back to the build time — the live site showed the same "Last updated" on every page regardless of when it actually changed. CI now checks out with `fetch-depth: 0`. Separately, `robots.txt` declares open Content-Signals (`search=yes, ai-input=yes, ai-train=yes`): WaveHouse is open source, so agents are welcome to read, index, answer from, and train on these docs. (A markdown-twin `Link` header — the other half of the agent-discoverability pass — is being upstreamed to [`cloudflare-md-router`](https://github.com/Wave-RF/cloudflare-md-router) instead, where the twin concept lives.) -- **Mermaid diagrams are click-to-zoom, with a top-level house style to author them vertically** (`docs/src/components/MermaidZoom.astro` (new), `docs/src/components/Footer.astro`, `docs/src/styles/global.css`, `AGENTS.md`): wide `flowchart LR` diagrams shrink to ~36% in the content column and are hard to read inline. Clicking any diagram now opens it in a lightbox — small diagrams scale up to fill the viewport (capped 2.5×), wide ones show at natural size with pan-scroll for full detail; Esc / backdrop / button closes, keyboard-focusable with Enter. View-transition-safe (one delegated `document` listener guarded by a window flag, re-enhanced on `astro:page-load`). The clone is re-id'd so its id-scoped inline `` itself could never clear), the phone Mermaid wrapper-scroller rule extends to `class`/`state` diagrams (they had the 540px no-JS min-width floor without the matching `overflow-x: auto`), and the search-dialog backdrop dropped a reintroduced hand-written `-webkit-backdrop-filter` twin (the same LightningCSS author-managed-prefix trap documented at the glass header — the build emits the twin itself). Bare issue references in prose became links (#97, #223, #224, #279). The mobile icon-only search trigger joins the menu button's circular control family (it rendered as an unstyled square), and external sidebar links (Contributing Guide, Discussions, …) carry a `↗` marker. +- **SDK docs split into six topic pages, with package-manager-synced install tabs** (`docs/src/content/docs/sdk/{index.mdx,queries.md,streaming.md,pipes.md,admin.md,reference.md}` (new), `docs/src/content/docs/sdk.md` (deleted), `docs/src/config/sidebar.ts`, `docs/src/components/Hero.astro`): the 806-line monolith becomes `/sdk` (overview: install, quick start, client setup, `Result`, section cards) plus `/sdk/{queries,streaming,pipes,admin,reference}`, under a nested "TypeScript SDK" sidebar group (which also keeps the llms.txt page ordering coherent — the LLM outputs read the sidebar). Topic-first URLs are the multi-language plan on record: when a second SDK language lands, the shared usage pages grow `` code tabs and each language gets its own setup/caveats page, so the topic URLs never churn. Installation became `` (pnpm / npm / yarn / bun / deno / CDN), and the landing hero's click-to-copy install chip now follows that synced choice — Starlight persists the selected tab label in `localStorage` under its syncKey, the hero script maps it to the matching one-liner (CDN and anything unknown keep the pnpm default). Every inbound link from other pages already pointed at bare `/sdk` (still the overview); the three intra-page `#streaming` anchors became `/sdk/streaming` links, and `starlight-links-validator` gates the rest. A follow-up adds an "Imports & Runtimes" section to the overview — import-line tabs synced to the same `pkg` choice (the CDN reader sees the esm.sh import everywhere, including a fully-tabbed Quick Start), Node CJS `require()` and the `WaveHouse` IIFE global documented from the shipped exports map, and a verified runtime-support note (streaming needs an `EventSource` polyfill in Node — the global is still flag-gated as of Node 26, checked empirically; non-streaming works on Node 18+). Each topic page points CDN readers at the section. - **Pre-push CI marker is tree-keyed, not commit-keyed** (`Makefile`, `.githooks/pre-push`, `.claude/hooks/agent-bash-gate.sh`, `AGENTS.md`). The `make ci → git add → git commit → git push` flow no longer false-positive-blocks — committing advances HEAD but doesn't change the tree CI validated. `make ci` writes `tmp/ci-passed-tree-` computed against a throwaway `GIT_INDEX_FILE` (so untracked files are captured without touching the real index); pre-push and the agent-bash-gate resolve each pushed ref's `^{tree}` and match. Skipped on CI runners (they don't push). `review-passed` stays commit-keyed — a new commit needs a new review. - **Worktrunk seeds new worktrees from main** (`.config/wt.toml`). Added `wt step copy-ignored` to `post-start` ahead of `make tools`, so `wt switch --create ` copies `.bin/` and the three `node_modules/` trees (~1 GB combined) from main instead of cold-starting `pnpm install --frozen-lockfile` × 3. `[step.copy-ignored] exclude` keeps `tmp/`, `data/`, and vitest's `coverage/` out. - **Claude PR review is now manual-only** (`.github/workflows/claude-review.yml`, `AGENTS.md`): dropped the `pull_request` trigger (was firing on opened/synchronize/reopened/ready_for_review) so the workflow only runs on `@claude` or `/review` comments from trusted reviewers, or `workflow_dispatch`. "Claude review" was never a required status check (the ruleset gates on `Admin approval` / `CI` / `PR housekeeping`), so this changes who pays the action budget — every push no longer triggers a review automatically; reviewers explicitly opt in via mention after they've looked at the diff themselves. Falls out: the workflow header's `pull_request_target` OIDC paragraph dropped (only applied to the auto-trigger path), concurrency's `cancel-in-progress` is now unconditionally `false` (manual triggers should always queue, never cancel an in-flight review), the workflow-level `if:` loses its `pull_request` branch + dependabot-author exclusion, and the gate-step `case` drops the `pull_request)` arm + its two `PR_EVENT_*` env vars (`gh pr view --jq .headRefOid` is hoisted out of the case since both remaining paths need it identically). The trust gate stays — it's *more* load-bearing under manual-only, since `issue_comment` runs with full repo secrets and a trusted reviewer commenting `@claude` on a fork PR would otherwise check out untrusted fork code with write tokens. AGENTS.md synced in three places: §"Review Response" step 6 (Claude moved from synchronize-rerun to PR-comment-mention), the review tooling reference table (Claude row now "Manual-only" with "No" in the re-runs column), and §"Repository Automation" §"Code review" entry (rewritten to lead with manual-only). diff --git a/docs/astro.config.mjs b/docs/astro.config.mjs index 3825ccc4..c738cb56 100644 --- a/docs/astro.config.mjs +++ b/docs/astro.config.mjs @@ -116,6 +116,14 @@ export default defineConfig({ lastUpdated: true, expressiveCode: { themes: ["github-dark", "github-light"], + // Long lines wrap with a hanging indent instead of growing a + // horizontal scrollbar — in a 45rem column, scroll-to-read code was + // the docs' biggest readability complaint. Authors can still opt a + // block out with ```lang wrap=false where alignment matters. + defaultProps: { + wrap: true, + preserveIndent: true, + }, styleOverrides: { // 0.75rem − 1px so the frame's OUTER corner lands at exactly 12px // (--radius-md, matching cards): EC adds the 1px border width to the diff --git a/docs/src/components/Footer.astro b/docs/src/components/Footer.astro index 65fc26f8..651eae2b 100644 --- a/docs/src/components/Footer.astro +++ b/docs/src/components/Footer.astro @@ -7,6 +7,8 @@ import EditLink from "virtual:starlight/components/EditLink"; import LastUpdated from "virtual:starlight/components/LastUpdated"; import Pagination from "virtual:starlight/components/Pagination"; import MermaidZoom from "./MermaidZoom.astro"; +import ReadingProgress from "./ReadingProgress.astro"; +import ScrollHints from "./ScrollHints.astro"; import WaveMark from "./WaveMark.astro"; const year = new Date().getFullYear(); @@ -24,6 +26,7 @@ const repo = "https://github.com/Wave-RF/WaveHouse"; { hasSidebar ? (
+
itself + // would never clear, since scroll events fire on the wrapper. + const SCROLLERS = ".expressive-code pre, .wh-scroll-x, .wh-zoom__stage"; + + function wrapScrollables() { + // Bare svgs in the content flow AND direct svg children of a + // .diagram-pair (rehype-mermaid sometimes unwraps the

) each get + // their OWN scroller — classing the pair container itself gave two + // diagrams one shared zoom chip and one shared natural width. + const bare = document.querySelectorAll( + ".sl-markdown-content > svg[aria-roledescription], .diagram-pair > svg[aria-roledescription]", + ); + for (const svg of bare) { + const div = document.createElement("div"); + div.className = "wh-scroll-x"; + svg.replaceWith(div); + div.append(svg); + } + // Surface each diagram's natural width (Mermaid stamps it as an inline + // max-width) as a custom property. global.css uses it as the min-width + // pin on phones only — pan at natural size beats 5px labels there — + // while desktop keeps scale-to-fit (an unconditional inline min-width + // here once forced wide diagrams into horizontal scrollers at every + // viewport). + for (const svg of document.querySelectorAll( + ".wh-scroll-x > svg[aria-roledescription], .sl-markdown-content :is(p, div) > svg[aria-roledescription]", + )) { + if (svg.style.maxWidth) { + svg.style.setProperty("--wh-natural-w", svg.style.maxWidth); + // The WRAPPER needs it too: its desktop breakout width is + // min(natural, pane) — fit-content can't measure a width:100% svg. + svg.parentElement?.style.setProperty("--wh-natural-w", svg.style.maxWidth); + } + } + // Paired diagrams already live in a

/

wrapper that global.css + // makes scrollable; give those the same class so they share the scrim + // and the touch zoom hint. + const wrapped = document.querySelectorAll( + ".sl-markdown-content :is(p, div) > svg[aria-roledescription]", + ); + for (const svg of wrapped) { + const parent = svg.parentElement; + if ( + parent && + !parent.classList.contains("wh-scroll-x") && + !parent.classList.contains("diagram-pair") + ) { + parent.classList.add("wh-scroll-x"); + } + } + // Surface each code frame's language for the corner badge — CSS attr() + // can only read from the element it styles, and EC stamps the language + // on the
 while the badge needs to live on the frame (inside the
+      // title bar for terminal/titled frames).
+      for (const pre of document.querySelectorAll(".expressive-code pre[data-language]")) {
+        const frame = pre.closest(".frame");
+        if (frame && !frame.hasAttribute("data-wh-lang")) {
+          frame.setAttribute("data-wh-lang", pre.dataset.language);
+        }
+      }
+      // Tables: a scroll WRAPPER lets the table itself use true auto layout
+      // (prose wraps first; the wrapper scrolls only when unbreakable columns
+      // genuinely can't fit). Without JS, the table's own display:block
+      // fallback in global.css still scrolls — it just doesn't wrap prose.
+      const tables = document.querySelectorAll(".sl-markdown-content table");
+      for (const t of tables) {
+        if (t.parentElement && t.parentElement.classList.contains("wh-table-wrap")) continue;
+        const div = document.createElement("div");
+        div.className = "wh-scroll-x wh-table-wrap";
+        t.replaceWith(div);
+        div.append(t);
+      }
+      tokenizeTableCode();
+    }
+
+    // Code in table cells must never split MID-token (`--url` reading as two
+    // flags) — but blanket nowrap also stopped multi-word code (JSON bodies,
+    // shell commands) from wrapping at its SPACES, which forced whole tables
+    // into scrollers. Split each plain-text code span into per-token nowrap
+    // s: wrapping happens between tokens only. Tokens with interior
+    // slashes (paths, endpoints) additionally get a  after each slash —
+    // the conventional URL break points (a  copies as nothing, so
+    // clipboard text is unchanged). The data attribute flips the CSS from the
+    // no-JS nowrap fallback to normal wrapping, and keeps this idempotent
+    // across view-transition reruns.
+    function tokenizeTableCode() {
+      for (const code of document.querySelectorAll(".wh-table-wrap code")) {
+        if (code.dataset.whTok || code.children.length) continue;
+        code.dataset.whTok = "1";
+        const frag = document.createDocumentFragment();
+        for (const part of code.textContent.split(/(\s+)/)) {
+          if (!part) continue;
+          if (/^\s/.test(part)) {
+            frag.append(part);
+            continue;
+          }
+          // Slash-segments each get their OWN nowrap span with the s
+          // between them in normal white-space — a  inside nowrap is
+          // suppressed, and a bare normal-flow token could still break at
+          // interior hyphens (the exact mid-token shred this prevents).
+          const bits = part.split(/(?<=\/)(?=.)/);
+          const host = document.createElement("span");
+          host.className = bits.length > 1 ? "wh-tok-path" : "wh-tok";
+          bits.forEach((b, i) => {
+            if (bits.length > 1) {
+              const seg = document.createElement("span");
+              seg.className = "wh-tok";
+              seg.textContent = b;
+              host.append(seg);
+            } else {
+              host.append(b);
+            }
+            if (i < bits.length - 1) host.append(document.createElement("wbr"));
+          });
+          frag.append(host);
+        }
+        code.replaceChildren(frag);
+      }
+    }
+
+    // 12px threshold, not ~0: fit-content widths and devicePixelRatio
+    // rounding leave phantom pixels of "overflow" on blocks with nothing
+    // real to scroll to — fading those reads as cut-off content that isn't
+    // there. Anything under ~a character width isn't worth hinting anyway.
+    function updateOne(el) {
+      // A WRAPPED code block can never meaningfully scroll sideways —
+      // pre-wrap + break-word guarantee every line fits. Whatever
+      // scrollWidth reports beyond clientWidth is hanging wrap-point
+      // whitespace or zoom/font-timing rounding, and this artifact class
+      // resisted several threshold bumps because it depends on exactly
+      // where lines wrap at the reader's width. Hard rule: no fades on
+      // pre.wrap, ever. (wrap=false blocks lack the class and keep their
+      // real scroll + fades.)
+      if (el.tagName === "PRE" && el.classList.contains("wrap")) {
+        el.removeAttribute("data-wh-more");
+        el.removeAttribute("data-wh-more-start");
+        return;
+      }
+      const end = el.scrollWidth - el.clientWidth - el.scrollLeft > 12;
+      const start = el.scrollLeft > 12;
+      el.toggleAttribute("data-wh-more", end);
+      el.toggleAttribute("data-wh-more-start", start);
+    }
+    function updateAll() {
+      document.querySelectorAll(SCROLLERS).forEach(updateOne);
+    }
+    // Tables earn their treatments by measurement, in two passes:
+    //
+    // Pass 1 (at prose width): a table that overflows or CROWDS there —
+    // ≥4 columns squeezed below ~8.5rem average wrap every other word —
+    // gets data-wh-wide, which on desktop moves it to the breakout track
+    // (wider, centered). A table whose prose wraps comfortably at 45rem
+    // stays at prose width: stretching its sentences to 60rem line lengths
+    // reads worse, not better.
+    //
+    // Pass 2 (re-measured at whatever width pass 1 granted): still
+    // overflowing or crowded → data-wh-scrolls, which enables the
+    // horizontal scroller and the per-column min-width floors (floors stop
+    // auto layout crushing off-screen columns, but applied to a fitting
+    // table they would CREATE the scroll).
+    function classifyTables() {
+      const CROWD_PX = 136; // below this avg column width, scroll beats wrapping
+      const TIGHT_PX = 200; // below this, a ≥4-col table is worth widening
+      const colsOf = (el) => {
+        const row = el.querySelector("tr");
+        return row ? row.cells.length : 0;
+      };
+      const overflowing = (el, slack) => el.scrollWidth - el.clientWidth > slack;
+      const wraps = [...document.querySelectorAll(".wh-table-wrap")];
+      wraps.forEach((el) => {
+        el.removeAttribute("data-wh-scrolls");
+        el.removeAttribute("data-wh-wide");
+      });
+      // Natural (max-content) width per table, measured in one batched
+      // reflow: a table whose natural width fits the breakout track widens
+      // to EXACTLY that width and stops wrapping entirely (feature matrices)
+      // — while prose-heavy tables (whose natural width is a full unwrapped
+      // sentence) don't widen just to stretch their line lengths.
+      const naturals = new Map();
+      wraps.forEach((el) => {
+        const t = el.querySelector("table");
+        if (t) t.style.width = "max-content";
+      });
+      wraps.forEach((el) => {
+        const t = el.querySelector("table");
+        if (t) naturals.set(el, t.getBoundingClientRect().width);
+      });
+      wraps.forEach((el) => {
+        const t = el.querySelector("table");
+        if (t) t.style.width = "";
+      });
+      const md = document.querySelector(".sl-markdown-content");
+      // Mirror the CSS cap: min(calc(100% - 2.5rem), 68rem).
+      const avail = md ? Math.min(md.clientWidth - 40, 1088) : 0;
+      const wide = wraps.filter((el) => {
+        const cols = colsOf(el);
+        const natural = naturals.get(el) ?? 0;
+        // The 1.3 factor: a table whose natural width is CLOSE to the
+        // breakout track still wins from widening (a touch of wrapping at
+        // full width beats heavy wrapping at prose width) — while genuinely
+        // prose-heavy tables (natural = whole unwrapped sentences, several
+        // times the track) stay at the reading measure.
+        return (
+          overflowing(el, 1) ||
+          (cols >= 4 && el.clientWidth / cols < TIGHT_PX) ||
+          (natural > el.clientWidth + 8 && natural <= avail * 1.3)
+        );
+      });
+      wide.forEach((el) => el.setAttribute("data-wh-wide", ""));
+      // Re-measure at the granted width: still overflowing or crowded →
+      // scroller + column floors.
+      wide
+        .filter((el) => {
+          const cols = colsOf(el);
+          return overflowing(el, 8) || (cols >= 4 && el.clientWidth / cols < CROWD_PX);
+        })
+        .forEach((el) => el.setAttribute("data-wh-scrolls", ""));
+    }
+    // Code blocks earn breakout the same way tables do: widening is only
+    // worth it when it makes wrapping go AWAY. A block whose longest line
+    // fits the wide track grows to exactly that line (no wrap, no scroll);
+    // one whose longest line exceeds even the full pane would wrap anyway —
+    // stretching it just strands a few words on the continuation line, so
+    // it keeps the prose width. wrap=false blocks are the exception: they
+    // scroll instead of wrapping, so the full pane minimizes the scroll.
+    // Code-ONLY tab groups (the YAML/JSON/env switchers) classify as a unit:
+    // one panel can't break out alone — the docked tablist and every sibling
+    // panel share the frame. All panels are measured (inactive ones are
+    // display:none, so they get an inline display:block for the batched
+    // read; layout never reaches paint between write and restore) and the
+    // group earns data-wh-wide when any panel would. Groups with prose in a
+    // panel are left alone: widening would stretch sentence line lengths.
+    function classifyCode() {
+      const md = document.querySelector(".sl-markdown-content");
+      if (!md) return;
+      const blocks = [...document.querySelectorAll(".sl-markdown-content > .expressive-code")];
+      const groups = [...document.querySelectorAll(".sl-markdown-content > starlight-tabs")]
+        .map((el) => ({ el, panels: [...el.querySelectorAll(":scope > [role='tabpanel']")] }))
+        .filter(
+          (g) =>
+            g.panels.length &&
+            g.panels.every(
+              (p) =>
+                p.children.length === 1 &&
+                p.firstElementChild.classList.contains("expressive-code"),
+            ),
+        );
+      if (!blocks.length && !groups.length) return;
+      blocks.forEach((b) => b.removeAttribute("data-wh-wide"));
+      groups.forEach((g) => {
+        g.el.removeAttribute("data-wh-wide");
+        g.el.style.removeProperty("--wh-natural-w");
+        g.pres = g.panels.map((p) => p.querySelector("pre"));
+        g.hidden = g.panels.filter((p) => p.hasAttribute("hidden"));
+      });
+      const pres = blocks.map((b) => b.querySelector("pre"));
+      // Batched natural-width measurement (write-all → read-all → restore).
+      groups.forEach((g) =>
+        g.hidden.forEach((p) => {
+          // Inline display beats the UA [hidden] rule for the measurement.
+          p.style.display = "block";
+        }),
+      );
+      // EC's wrapping lives on the DESCENDANT code spans (`pre.wrap .ec-line
+      // .code { white-space: pre-wrap }`), so an inline white-space on the
+      // pre never unwraps anything — the wrap class itself comes off for the
+      // measurement. Restored before the classification loops (which test
+      // for it) and before anything paints.
+      const allPres = pres.concat(groups.flatMap((g) => g.pres));
+      const wrapped = allPres.filter((p) => p && p.classList.contains("wrap"));
+      wrapped.forEach((p) => p.classList.remove("wrap"));
+      const naturals = pres.map((p) => (p ? p.scrollWidth : 0));
+      groups.forEach((g) => {
+        g.naturals = g.pres.map((p, i) => {
+          if (!p) return 0;
+          // EC's frame border sits outside pre.scrollWidth but inside the
+          // group's fit-content width — fold it into the pin, or the frame
+          // breathes by the border width when switching to a narrower tab.
+          const chrome = g.panels[i].firstElementChild.offsetWidth - p.clientWidth;
+          return p.scrollWidth + chrome;
+        });
+        g.width = g.el.clientWidth;
+      });
+      wrapped.forEach((p) => p.classList.add("wrap"));
+      groups.forEach((g) => g.hidden.forEach((p) => p.style.removeProperty("display")));
+      const avail = md.clientWidth - 8;
+      blocks.forEach((b, i) => {
+        const p = pres[i];
+        if (!p) return;
+        const fitsWide = naturals[i] <= avail;
+        const needsMore = naturals[i] > b.clientWidth + 8;
+        if (needsMore && (fitsWide || !p.classList.contains("wrap"))) {
+          b.setAttribute("data-wh-wide", "");
+        }
+      });
+      groups.forEach((g) => {
+        const earns = g.pres.some((p, i) => {
+          if (!p) return false;
+          const fitsWide = g.naturals[i] <= avail;
+          const needsMore = g.naturals[i] > g.width + 8;
+          return needsMore && (fitsWide || !p.classList.contains("wrap"));
+        });
+        if (!earns) return;
+        g.el.setAttribute("data-wh-wide", "");
+        // Pin the group to its WIDEST panel so the frame keeps one stable
+        // width across tab switches instead of resizing per panel.
+        g.el.style.setProperty(
+          "--wh-natural-w",
+          `${Math.ceil(Math.min(Math.max(...g.naturals), avail))}px`,
+        );
+      });
+    }
+
+    function init() {
+      wrapScrollables();
+      classifyCode();
+      classifyTables();
+      updateAll();
+    }
+
+    init();
+    if (window.__whScrollHintsBound) return;
+    window.__whScrollHintsBound = true;
+    document.addEventListener("astro:page-load", init);
+    window.addEventListener(
+      "resize",
+      () =>
+        requestAnimationFrame(() => {
+          classifyCode();
+          classifyTables();
+          updateAll();
+        }),
+      { passive: true },
+    );
+    // Web fonts change every measured width — re-derive once they land, so
+    // a classification made against fallback-font metrics can't go stale.
+    if (document.fonts && document.fonts.ready) {
+      document.fonts.ready.then(() =>
+        requestAnimationFrame(() => {
+          classifyCode();
+          classifyTables();
+          updateAll();
+        }),
+      );
+    }
+    // A tab switch reveals a pre that was display:none (unmeasurable) when
+    // fades were last stamped — refresh once the swap has happened. Click
+    // and keydown both: Starlight's tablist also switches on arrow keys.
+    const onTabSwitch = (e) => {
+      if (e.target instanceof Element && e.target.closest("[role='tab']")) {
+        requestAnimationFrame(updateAll);
+      }
+    };
+    document.addEventListener("click", onTabSwitch);
+    document.addEventListener("keydown", onTabSwitch);
+    // Element scroll events don't bubble, but they do capture.
+    document.addEventListener(
+      "scroll",
+      (e) => {
+        const t = e.target;
+        if (t instanceof Element && t.matches && t.matches(SCROLLERS)) {
+          updateOne(t);
+        }
+      },
+      { capture: true, passive: true },
+    );
+  })();
+
+
+
diff --git a/docs/src/config/sidebar.ts b/docs/src/config/sidebar.ts
index d7a66b67..8e8da9de 100644
--- a/docs/src/config/sidebar.ts
+++ b/docs/src/config/sidebar.ts
@@ -16,8 +16,29 @@ export const sidebar: StarlightUserConfig["sidebar"] = [
       { label: "Ingest Pipeline", slug: "ingest-pipeline" },
       { label: "Access Control", slug: "access-control" },
       { label: "Named Pipes", slug: "pipes" },
+    ],
+  },
+  {
+    // Lookup material, not narrative — its own group so "I need the exact
+    // endpoint/method" readers don't scan the conceptual guides for it.
+    label: "Reference",
+    items: [
       { label: "API Reference", slug: "api" },
-      { label: "TypeScript SDK", slug: "sdk" },
+      {
+        // Topic-first SDK pages: when a second SDK language lands, these
+        // shared usage pages grow  code tabs and each
+        // language gets its own setup/caveats page — the topic URLs never
+        // churn (decision in PR #313).
+        label: "TypeScript SDK",
+        items: [
+          { label: "Overview", slug: "sdk" },
+          { label: "Queries", slug: "sdk/queries" },
+          { label: "Streaming & Live Queries", slug: "sdk/streaming" },
+          { label: "Pipes", slug: "sdk/pipes" },
+          { label: "Admin & System", slug: "sdk/admin" },
+          { label: "Reference & CLI", slug: "sdk/reference" },
+        ],
+      },
     ],
   },
   {
diff --git a/docs/src/content/docs/404.md b/docs/src/content/docs/404.md
index a4ab8fae..36e006ca 100644
--- a/docs/src/content/docs/404.md
+++ b/docs/src/content/docs/404.md
@@ -1,5 +1,5 @@
 ---
-title: "Page Not Found"
+title: "Page not found"
 description: "The page you were looking for doesn't exist."
 template: splash
 editUrl: false
@@ -11,11 +11,39 @@ head:
       content: noindex
 ---
 
-The page you were looking for isn't here. A few common starting points:
+
 
-- **[Getting Started](/getting-started)** — install, run, and ingest your first event
-- **[Architecture](/architecture)** — how the pieces fit together
-- **[API Reference](/api)** — endpoints, payloads, and error semantics
-- **[TypeScript SDK](/sdk)** — typed client for browser and Node
-
-If you followed a link from elsewhere and think it should have worked, please [file an issue](https://github.com/Wave-RF/WaveHouse/issues/new).
+
+ + +

Signal lost

+

There's no page at this address — the link may be stale, or the page may have moved. One of these will get you back on the air:

+ +

Followed a link that should have worked? File an issue — broken links are bugs.

+
diff --git a/docs/src/content/docs/access-control.md b/docs/src/content/docs/access-control.mdx similarity index 79% rename from docs/src/content/docs/access-control.md rename to docs/src/content/docs/access-control.mdx index 34717847..c58cb0dd 100644 --- a/docs/src/content/docs/access-control.md +++ b/docs/src/content/docs/access-control.mdx @@ -5,6 +5,8 @@ sidebar: order: 5 --- +import { Tabs, TabItem } from "@astrojs/starlight/components"; + WaveHouse authorizes every request against a single **access control policy**: a per-table, per-role document that decides which columns a caller may see, which rows they may touch, what they may aggregate, and how much work a query may do. It is Hasura-style — column-level and row-level permissions with JWT claim templating — stored centrally and applied on every read, write, and live stream. This guide explains the whole model: where roles come from, how to write a policy, how each rule is enforced, and how the policy is loaded and changed at runtime. For the raw config knobs see [Configuration](/configuration#access-control-policy); for the HTTP surface see the [API reference](/api#authentication). Named query pipes have their own, simpler authorization model — see [Named Pipes](/pipes). @@ -51,7 +53,9 @@ For authorization, the behavior that matters is what happens with **no** role: t Setting `default_role` does nothing on its own; the role you name still needs entries under `tables` to grant any access. Think of it as "which role does an anonymous caller assume", not "what can anonymous callers do". -> **`default_role: admin` is a dev-only footgun.** Setting `default_role` equal to `admin_role` is permitted — it makes every unauthenticated request a full admin, including `/v1/admin/*`, which is handy for local development with no tokens. It is **never** for production. Every node that loads such a policy logs a loud `WARN` on startup and on every update. +:::caution[`default_role: admin` is a dev-only footgun] +Setting `default_role` equal to `admin_role` is permitted — it makes every unauthenticated request a full admin, including `/v1/admin/*`, which is handy for local development with no tokens. It is **never** for production. Every node that loads such a policy logs a loud `WARN` on startup and on every update. +::: ### `admin_role` — the privileged role @@ -62,7 +66,9 @@ Setting `default_role` does nothing on its own; the role you name still needs en There is no separate `service` role. To reach an admin endpoint or to read data beyond what `default_role` grants, present a valid token whose `role_claim` is the admin role (or another granted role). -> **A missing policy locks everyone out — including admin.** `admin_role` only grants access while a policy is loaded. Deleting the policy from the store sets it to `nil`, and a `nil` policy denies everyone, by design: an implicit admin grant must never re-open a deliberately-emptied deployment. Recovery is server-side — restore the [bootstrap file](#bootstrapping-and-the-policy-lifecycle) and reboot, not an "admin" token over HTTP. +:::caution[A missing policy locks everyone out — including admin] +`admin_role` only grants access while a policy is loaded. Deleting the policy from the store sets it to `nil`, and a `nil` policy denies everyone, by design: an implicit admin grant must never re-open a deliberately-emptied deployment. Recovery is server-side — restore the [bootstrap file](#bootstrapping-and-the-policy-lifecycle) and reboot, not an "admin" token over HTTP. +::: ## Anatomy of a policy @@ -76,6 +82,8 @@ A policy is a YAML or JSON document with three top-level fields: Each table entry holds permissions for two operations — `select` (reads) and `insert` (writes) — and each operation maps **role → permissions**: + + ```yaml tables: events: # ClickHouse table name @@ -86,6 +94,28 @@ tables: writer: # role → what "writer" may insert allow_columns: ["page", "score", "user_id"] ``` + + +```json +{ + "tables": { + "events": { + "select": { + "viewer": { + "allow_columns": ["page", "score"] + } + }, + "insert": { + "writer": { + "allow_columns": ["page", "score", "user_id"] + } + } + } + } +} +``` + + A role with no entry under the operation it is attempting is denied. The set of permission fields below applies to both `select` and `insert` entries, though some (row `filter`, aggregations, limits) only take effect on the read path and some (`check`) only on the write path — the [enforcement table](#where-each-rule-is-enforced) maps which is which. @@ -93,12 +123,28 @@ A role with no entry under the operation it is attempting is denied. The set of `allow_columns` and `deny_columns` control which columns a role can read (on `select`) or write (on `insert`). + + ```yaml select: viewer: allow_columns: ["page", "button", "received_timestamp"] deny_columns: ["user_email", "ip_address"] ``` + + +```json +{ + "select": { + "viewer": { + "allow_columns": ["page", "button", "received_timestamp"], + "deny_columns": ["user_email", "ip_address"] + } + } +} +``` + + The rules, in order: @@ -112,6 +158,8 @@ On a structured query (`POST /v1/query?table={table}`) the allowlist is a **hard `filter` restricts *which rows* a role can read by injecting a `WHERE` clause into the generated SQL. Each entry maps a column to a comparison whose value is usually a **JWT claim template**: + + ```yaml select: viewer: @@ -119,6 +167,23 @@ select: tenant_id: _eq: "{{ jwt.app_metadata.tenant_id }}" ``` + + +```json +{ + "select": { + "viewer": { + "filter": { + "tenant_id": { + "_eq": "{{ jwt.app_metadata.tenant_id }}" + } + } + } + } +} +``` + + This produces `WHERE (tenant_id = ?)` with the caller's `app_metadata.tenant_id` claim bound as the parameter — so a `viewer` only ever sees rows for their own tenant, and the value comes from the signed token, not from anything the client sends. @@ -131,7 +196,7 @@ Supported comparison operators: | `_gt` | `col > ?` | greater than | | `_lt` | `col < ?` | less than | -> The policy `Filter` schema also accepts `_in`, but it is **not yet enforced** (tracked in #224) — an `_in` clause is silently ignored (no predicate is produced), so stick to the operators above. +> The policy `Filter` schema also accepts `_in`, but it is **not yet enforced** (tracked in [#224](https://github.com/Wave-RF/WaveHouse/issues/224)) — an `_in` clause is silently ignored (no predicate is produced), so stick to the operators above. Multiple columns (and multiple operators on one column) are combined with `AND`. The policy-derived predicate is ANDed with whatever filters the caller's own query supplies, so a caller can never widen their row visibility past the policy. @@ -148,6 +213,8 @@ Values are always bound as SQL **parameters**, never concatenated into the query `check` is the write-path counterpart to `filter`: it constrains the *values* a role may insert, again via claim templates. It only supports `_eq`. + + ```yaml insert: writer: @@ -158,6 +225,27 @@ insert: tenant_id: _eq: "{{ jwt.app_metadata.tenant_id }}" ``` + + +```json +{ + "insert": { + "writer": { + "allow_columns": ["page", "score", "user_id", "tenant_id"], + "check": { + "user_id": { + "_eq": "{{ jwt.sub }}" + }, + "tenant_id": { + "_eq": "{{ jwt.app_metadata.tenant_id }}" + } + } + } + } +} +``` + + For each checked column, on `POST /v1/ingest?table={table}`: @@ -170,13 +258,33 @@ This pairs naturally with a matching `filter` on the `select` side: `check` stam `allowed_aggregations` and `denied_aggregations` restrict which aggregation functions a role may use in a structured query (`count`, `sum`, `avg`, `quantile`, …). Matching is case-insensitive and follows the same precedence as columns: + + ```yaml select: viewer: - denied_aggregations: ["quantile", "median"] # blocklist: everything else is allowed + # blocklist: everything else is allowed + denied_aggregations: ["quantile", "median"] analyst: - allowed_aggregations: ["count", "sum", "avg"] # allowlist: only these three + # allowlist: only these three + allowed_aggregations: ["count", "sum", "avg"] ``` + + +```json +{ + "select": { + "viewer": { + "denied_aggregations": ["quantile", "median"] + }, + "analyst": { + "allowed_aggregations": ["count", "sum", "avg"] + } + } +} +``` + + - `denied_aggregations` always wins. - An empty `allowed_aggregations` means all non-denied functions are allowed; a non-empty list is an allowlist. @@ -192,12 +300,28 @@ Two fields cap the cost of a single structured query. Both must be non-negative; | `max_rows` | Caps the query's `LIMIT`. If the caller asks for more (or omits a limit), the result is clamped to `max_rows`. | | `max_execution_time_ms` | Caps the ClickHouse execution timeout for the query, applied as the *minimum* of this value and the server's `clickhouse.query_timeout`. | + + ```yaml select: viewer: max_rows: 1000 max_execution_time_ms: 5000 ``` + + +```json +{ + "select": { + "viewer": { + "max_rows": 1000, + "max_execution_time_ms": 5000 + } + } +} +``` + + These bound a role's blast radius on the cached read path. They do not apply to raw admin SQL, which is unbounded by design (other than the 64 MiB response cap noted in the [API reference](/api)). @@ -213,7 +337,9 @@ The same policy drives every data path, but not every field is meaningful on eve | Raw SQL | `POST /v1/admin/query` | `admin_role` only — no per-statement policy; the role gate is the entire authorization story | | Named pipe | `GET/POST /v1/pipes/{name}` | not the policy engine — per-pipe `allowed_roles`; see [Named Pipes](/pipes) | -> **Live streams enforce access, not row filters.** SSE subscribers are checked for table-level `select` permission and have denied columns stripped from each event, but the row-level `filter` predicates and `max_rows`/`max_execution_time_ms` limits are a property of the SQL query path and are **not** applied to the live event stream. If a role must never observe another tenant's rows in real time, don't grant it stream access to a shared table — scope the data at the table level. +:::caution[Live streams enforce access, not row filters] +SSE subscribers are checked for table-level `select` permission and have denied columns stripped from each event, but the row-level `filter` predicates and `max_rows`/`max_execution_time_ms` limits are a property of the SQL query path and are **not** applied to the live event stream. If a role must never observe another tenant's rows in real time, don't grant it stream access to a shared table — scope the data at the table level. +::: ## Managing the policy @@ -246,21 +372,36 @@ NATS KV is the **source of truth** for the live policy. A file is only ever a *s - **When `policy.file_path` is set, the file must exist and parse**, or WaveHouse refuses to boot. That turns a typo or a missing mount into a loud failure instead of a silent fail-closed deployment that denies everything. - **When `policy.file_path` is empty and KV is empty**, the store comes up with no policy — every request is denied (logged loudly) until you seed one via `PUT /v1/admin/policy`. +In `config.yaml` (or the matching environment variable): + + + ```yaml -# config.yaml policy: - file_path: /etc/wavehouse/policy.yaml # seed on first boot only; KV wins thereafter + # seed on first boot only; KV wins thereafter + file_path: /etc/wavehouse/policy.yaml ``` + + +```ini +WH_POLICY_FILE_PATH=/etc/wavehouse/policy.yaml +``` + + Because a deleted policy is a total lockout (including admin), treat the bootstrap file as your recovery path: keep it in version control, mount it read-only, and you can always restore a known-good policy by clearing KV state and rebooting. See [Configuration — Access Control Policy](/configuration#access-control-policy) for the exact knobs. ## A complete example -A multi-tenant analytics deployment: anonymous callers get nothing, `viewer` reads its own tenant's rows with sensitive columns masked and expensive aggregations blocked, `writer` ingests events that are stamped with the caller's identity, and `admin` is unrestricted. +A multi-tenant analytics deployment: anonymous callers get nothing, `viewer` reads its own tenant's rows with sensitive columns masked and expensive aggregations blocked, `writer` ingests events that are stamped with the caller's identity, and `admin` is unrestricted. Save it as your `policy.file_path` bootstrap file (`policy.yaml` or `policy.json` — both parse identically), or `PUT` it to `/v1/admin/policy`: + + ```yaml -admin_role: admin # full access + the /v1/admin/* gate (this is also the default) -default_role: "" # "" = closed: a request with no/!valid token is denied +# admin: full access + the /v1/admin/* gate (this is also the default) +admin_role: admin +# "" = closed: a request with no/!valid token is denied +default_role: "" tables: events: @@ -285,6 +426,46 @@ tables: tenant_id: _eq: "{{ jwt.app_metadata.tenant_id }}" ``` + + +```json +{ + "admin_role": "admin", + "default_role": "", + "tables": { + "events": { + "select": { + "viewer": { + "deny_columns": ["user_email", "ip_address"], + "filter": { + "tenant_id": { + "_eq": "{{ jwt.app_metadata.tenant_id }}" + } + }, + "denied_aggregations": ["quantile", "median"], + "max_rows": 1000, + "max_execution_time_ms": 5000 + } + }, + "insert": { + "writer": { + "allow_columns": ["event_name", "page", "score", "user_id", "tenant_id"], + "check": { + "user_id": { + "_eq": "{{ jwt.sub }}" + }, + "tenant_id": { + "_eq": "{{ jwt.app_metadata.tenant_id }}" + } + } + } + } + } + } +} +``` + + With this policy loaded: diff --git a/docs/src/content/docs/api.md b/docs/src/content/docs/api.md index 5f9212a4..1a71d3ff 100644 --- a/docs/src/content/docs/api.md +++ b/docs/src/content/docs/api.md @@ -68,7 +68,9 @@ Historically some error paths defaulted to `text/plain` because they were emitte The per-endpoint error tables below list the bodies you can expect for each status code; the `Content-Type` and `X-Content-Type-Options` headers above apply uniformly and are not repeated. -> **Caveat — streaming / partial-write responses.** For SSE, streaming endpoints, or any handler that has already started writing the response, a later panic is recovered and logged server-side but no JSON 500 body is written — once headers are flushed, replacing them would corrupt the stream. Clients consuming streams should treat connection termination or truncated output as the failure signal in those cases. +:::caution[Streaming / partial-write responses] +For SSE, streaming endpoints, or any handler that has already started writing the response, a later panic is recovered and logged server-side but no JSON 500 body is written — once headers are flushed, replacing them would corrupt the stream. Clients consuming streams should treat connection termination or truncated output as the failure signal in those cases. +::: ## Endpoints @@ -87,7 +89,10 @@ Returns `200 OK` once the gateway has discovered ClickHouse table schemas at lea **Response (boot-degraded):** ```json -{"status": "degraded", "error": "schema discovery: dial tcp 127.0.0.1:9000: connect: connection refused"} +{ + "status": "degraded", + "error": "schema discovery: dial tcp 127.0.0.1:9000: connect: connection refused" +} ``` Status code: `503 Service Unavailable` @@ -174,9 +179,11 @@ The inbound request body is capped at 16 MiB; a body over the cap is rejected wi The `{table}` URL query must match a table that exists in ClickHouse. WaveHouse discovers table schemas on startup and refreshes them periodically. -> **Insert-only.** The ingest pipeline accepts only inserts. All other mutations — `DELETE`, `UPDATE`, `TRUNCATE`, `DROP`, `ALTER`, `REPLACE`, etc. — must be issued through [`POST /v1/admin/query`](#post-v1adminquery--query-clickhouse), which is restricted to the admin role (`admin_role`, the same gate as the rest of `/v1/admin/*`). -> -> The policy engine authorizes mutations by inspecting the columns being written. That works for inserts but not for predicate-driven mutations like `DELETE … WHERE` — there's no way to prove the predicate matches only rows the caller is allowed to touch. Routing those statements through the admin-gated raw-SQL surface keeps the policy contract honest. +:::note[Insert-only] +The ingest pipeline accepts only inserts. All other mutations — `DELETE`, `UPDATE`, `TRUNCATE`, `DROP`, `ALTER`, `REPLACE`, etc. — must be issued through [`POST /v1/admin/query`](#post-v1adminquery--query-clickhouse), which is restricted to the admin role (`admin_role`, the same gate as the rest of `/v1/admin/*`). + +The policy engine authorizes mutations by inspecting the columns being written. That works for inserts but not for predicate-driven mutations like `DELETE … WHERE` — there's no way to prove the predicate matches only rows the caller is allowed to touch. Routing those statements through the admin-gated raw-SQL surface keeps the policy contract honest. +::: **Request:** @@ -297,7 +304,9 @@ A `200` is returned whenever the body was read and the records were processed | 500 | `{"error":"publish failed"}` / `{"error":"dedupe failed"}` | Message-queue or dedup-backend failure mid-batch | | 503 | `{"error":"service unavailable"}` | NATS JetStream full (backpressure) mid-batch; includes `Retry-After: 30` | -> **At-least-once on retry.** A batch aborted partway (a `503`/`500`, or a JSON-array syntax error, after some leading records were already published) re-publishes those leading records when the whole batch is retried. Enable deduplication if duplicate suppression matters — this is the same at-least-once property the single-object path already has (the SDK retries both on `503`). +:::caution[At-least-once on retry] +A batch aborted partway (a `503`/`500`, or a JSON-array syntax error, after some leading records were already published) re-publishes those leading records when the whole batch is retried. Enable deduplication if duplicate suppression matters — this is the same at-least-once property the single-object path already has (the SDK retries both on `503`). +::: **curl example (JSON array):** @@ -321,13 +330,19 @@ curl -X POST "http://localhost:8080/v1/ingest?table=clicks" \ Executes a SQL statement directly against ClickHouse. **WaveHouse proxies the SQL string verbatim to ClickHouse's HTTP interface** — any statement ClickHouse accepts works, including arbitrary DDL/DML/SYSTEM verbs and inline FORMAT directives. Multi-statement input (`SELECT 1; TRUNCATE t`) also works on recent ClickHouse versions where multi-query is enabled by default; older or restrictively-configured servers may reject the second statement with a clear error. Read queries return a JSON array of result rows; mutations/DDL return HTTP 200 with `[]` on success. DateTime columns are ISO-8601 formatted via the upstream `date_time_output_format=iso` setting; other types are returned as ClickHouse renders them under `FORMAT JSON`. -> **Inline `FORMAT` overrides the JSON envelope.** ClickHouse's inline `FORMAT` clause (e.g. `SELECT 1 FORMAT CSV` or `… FORMAT Pretty`) takes precedence over the URL-level `default_format=JSON` setting. When the SQL contains an explicit `FORMAT`, the proxy forwards ClickHouse's raw response body (CSV, Pretty, TSV, …) and passes through the upstream `Content-Type` header — `text/csv`, `text/tab-separated-values`, etc. — so consumers see the right MIME type. The "extract the `data` array" behavior only applies when ClickHouse returned the `FORMAT JSON` envelope, which is the default. -> -> **64 MiB response cap.** The proxy buffers the upstream response in memory before forwarding (no row-streaming yet), so a `SELECT *` from a large table can pin RAM on the API server. To avoid an admin OOMing themselves, responses larger than 64 MiB return 502 with a `clickhouse response exceeded N bytes` error. Narrow the query with `LIMIT`, or use a streaming client outside WaveHouse that talks to ClickHouse directly (the standard escape hatch — the same admin credentials work). +:::note[Inline `FORMAT` overrides the JSON envelope] +ClickHouse's inline `FORMAT` clause (e.g. `SELECT 1 FORMAT CSV` or `… FORMAT Pretty`) takes precedence over the URL-level `default_format=JSON` setting. When the SQL contains an explicit `FORMAT`, the proxy forwards ClickHouse's raw response body (CSV, Pretty, TSV, …) and passes through the upstream `Content-Type` header — `text/csv`, `text/tab-separated-values`, etc. — so consumers see the right MIME type. The "extract the `data` array" behavior only applies when ClickHouse returned the `FORMAT JSON` envelope, which is the default. +::: + +:::caution[64 MiB response cap] +The proxy buffers the upstream response in memory before forwarding (no row-streaming yet), so a `SELECT *` from a large table can pin RAM on the API server. To avoid an admin OOMing themselves, responses larger than 64 MiB return 502 with a `clickhouse response exceeded N bytes` error. Narrow the query with `LIMIT`, or use a streaming client outside WaveHouse that talks to ClickHouse directly (the standard escape hatch — the same admin credentials work). +::: This endpoint **does not cache, does not singleflight, and emits `Cache-Control: no-store`** — every request goes straight to ClickHouse, mutation or read, and downstream HTTP caches are explicitly told not to store the response. Raw SQL is an admin escape hatch with infrequent, ad-hoc traffic, so the L1/singleflight machinery would only add complexity without a real hit-rate win. Use [`POST /v1/query?table={table}`](#post-v1querytabletable--structured-query) or [`GET/POST /v1/pipes/{name}`](#getpost-v1pipesname--execute-named-pipe) for the cached read paths (dashboards, high-QPS clients, etc.) — both share an in-process L1 (Ristretto) with singleflight coalescing. -> **Admin only.** The route is mounted under `/v1/admin/*`, behind the `RequireAdmin` gate: only a caller whose JWT role equals the policy `admin_role` (`"admin"` by default) may use it. A request with no/invalid token resolves to the `default_role` (not the admin role unless `default_role` is deliberately set to it — a loudly-warned dev-only setting) and is rejected. Raw SQL has no per-statement scope check (a full SQL parser would be needed to authorize predicates), so the role gate is the entire authorization story, shared with the rest of `/v1/admin/*` (policy CRUD, pipes CRUD). The normal surfaces for non-admin callers are `POST /v1/ingest?table={table}` for writes, `POST /v1/query?table={table}` for structured reads, and `GET/POST /v1/pipes/{name}` for pre-defined queries — none of which expose raw SQL. +:::note[Admin only] +The route is mounted under `/v1/admin/*`, behind the `RequireAdmin` gate: only a caller whose JWT role equals the policy `admin_role` (`"admin"` by default) may use it. A request with no/invalid token resolves to the `default_role` (not the admin role unless `default_role` is deliberately set to it — a loudly-warned dev-only setting) and is rejected. Raw SQL has no per-statement scope check (a full SQL parser would be needed to authorize predicates), so the role gate is the entire authorization story, shared with the rest of `/v1/admin/*` (policy CRUD, pipes CRUD). The normal surfaces for non-admin callers are `POST /v1/ingest?table={table}` for writes, `POST /v1/query?table={table}` for structured reads, and `GET/POST /v1/pipes/{name}` for pre-defined queries — none of which expose raw SQL. +::: `/v1/admin/query` is the only sanctioned surface for non-insert mutations (the ingest pipeline is insert-only). Granting raw-SQL access to a non-admin role via the policy engine is no longer supported: authenticate with the admin role (`admin_role`). @@ -343,7 +358,9 @@ This endpoint **does not cache, does not singleflight, and emits `Cache-Control: | ----- | ---- | -------- | ----------- | | `sql` | string | Yes | SQL forwarded verbatim to ClickHouse's HTTP interface. | -> **No parameter binding on this endpoint (yet).** The earlier handler accepted a `params` array bound to `?` placeholders; the HTTP proxy doesn't. ClickHouse's native named-param syntax (`WHERE id = {id:UInt32}` with `param_id=42` on the URL query string) is *not* forwarded today either — the proxy only sets `default_format`, `date_time_output_format`, and `database` on the upstream URL, and the request body is `{"sql": "..."}` with no escape hatch for query-string params. The current contract is "send raw SQL, get rows back": inline literals into the SQL for now. For safe binding from user-supplied input, use the structured query endpoint (`POST /v1/query?table={table}`) — that's its job. +:::note[No parameter binding on this endpoint (yet)] +The earlier handler accepted a `params` array bound to `?` placeholders; the HTTP proxy doesn't. ClickHouse's native named-param syntax (`WHERE id = {id:UInt32}` with `param_id=42` on the URL query string) is *not* forwarded today either — the proxy only sets `default_format`, `date_time_output_format`, and `database` on the upstream URL, and the request body is `{"sql": "..."}` with no escape hatch for query-string params. The current contract is "send raw SQL, get rows back": inline literals into the SQL for now. For safe binding from user-supplied input, use the structured query endpoint (`POST /v1/query?table={table}`) — that's its job. +::: **Response:** @@ -387,7 +404,9 @@ curl -X POST http://localhost:8080/v1/admin/query \ Executes a type-safe structured query against a table. The query AST is validated against the schema and converted to parameterized SQL. Permissions from the access control policy are enforced (column filtering, row-level security, aggregation restrictions). -> **The column allowlist is a hard cap on every clause.** Every column the query references — in `columns`, an aggregation argument, `filters`, `group_by`, `order_by`, or `time_range` — must be permitted by the role's `allow_columns`/`deny_columns`, or the request is rejected with `403 column "x" not allowed`. A full-row read is requested with `"select_all": true` (expanded to the columns the role may read — never a raw `SELECT *`); **omitting `columns` returns nothing**, so a hidden column never leaks by being left out, grouped on, or filtered on. See [Access control → Column permissions](/access-control#column-permissions). +:::note[The column allowlist is a hard cap on every clause] +Every column the query references — in `columns`, an aggregation argument, `filters`, `group_by`, `order_by`, or `time_range` — must be permitted by the role's `allow_columns`/`deny_columns`, or the request is rejected with `403 column "x" not allowed`. A full-row read is requested with `"select_all": true` (expanded to the columns the role may read — never a raw `SELECT *`); **omitting `columns` returns nothing**, so a hidden column never leaks by being left out, grouped on, or filtered on. See [Access control → Column permissions](/access-control#column-permissions). +::: **Request:** @@ -422,7 +441,9 @@ Executes a type-safe structured query against a table. The query AST is validate | `limit` | int | No | Max rows. Omitted or above 10,000 → silently capped at 10,000 (`DefaultMaxRows`); a policy `max_rows` can lower it further (see [Access Control](/access-control#resource-limits)). | | `time_range` | object | No | Time window (`column`, `since`, `until`). `since`/`until` accept RFC3339 or Go-duration relative values ("1h", "30m", "7d", "2w" — day and week suffixes expand to hours). Relative values mean that long *ago*. The window applies only when `column` and `since` are set — an `until` without `since` is ignored. | -> **Identifier names.** Table, column, and alias names may contain any characters ClickHouse accepts — dots, spaces, unicode, reserved keywords — because every identifier is backtick-quoted automatically. The one exception is a name containing a literal `?`, which is rejected with `400` (a clickhouse-go positional-binder limitation tracked in [#279](https://github.com/Wave-RF/WaveHouse/issues/279)). +:::note[Identifier names] +Table, column, and alias names may contain any characters ClickHouse accepts — dots, spaces, unicode, reserved keywords — because every identifier is backtick-quoted automatically. The one exception is a name containing a literal `?`, which is rejected with `400` (a clickhouse-go positional-binder limitation tracked in [#279](https://github.com/Wave-RF/WaveHouse/issues/279)). +::: **Response:** @@ -519,7 +540,9 @@ curl -N "http://localhost:8080/v1/stream?table=clicks&since=2026-03-24T11:00:00Z Returns all discovered ClickHouse table schemas. -> **Admin only.** The schema and DLQ endpoints in this section require the `admin_role` (like [`/v1/admin/query`](#post-v1adminquery--query-clickhouse)); other callers get 401 (bad token) / 403. The quickstart's trial `public` role cannot call them. +:::note[Admin only] +The schema and DLQ endpoints in this section require the `admin_role` (like [`/v1/admin/query`](#post-v1adminquery--query-clickhouse)); other callers get 401 (bad token) / 403. The quickstart's trial `public` role cannot call them. +::: **Response:** diff --git a/docs/src/content/docs/architecture.md b/docs/src/content/docs/architecture.md index ab524ee3..b4ca503b 100644 --- a/docs/src/content/docs/architecture.md +++ b/docs/src/content/docs/architecture.md @@ -140,7 +140,7 @@ The package's design invariants — stdout always 100%, WARN+ERROR always export ### `query/` — Structured Query Engine - **ast.go** — `StructuredQuery` AST types: columns, aggregations, filters, group by, order by, limit, time range. -- **builder.go** — `Build()` converts AST to parameterized SQL. It is the single chokepoint that validates every referenced identifier against the schema **and** authorizes every column reference — projection, aggregation args, filters, group_by, order_by, time_range — against the role's column allowlist (the #223 hard cap). A full-row read is requested with `select_all`, which expands to the role's allowed columns rather than emitting a raw `SELECT *`; an omitted projection selects nothing, and `*` in `columns` is a literal column name. Every identifier is backtick-quoted via `internal/chsql` (`QuoteIdent`) so any ClickHouse-legal name is accepted — a name containing `?` is refused fail-closed (#279). `InjectPermissionFilters()` adds row-level security. `ApplyMaxRows()` enforces limits. Timestamp bucketing for cache optimization. +- **builder.go** — `Build()` converts AST to parameterized SQL. It is the single chokepoint that validates every referenced identifier against the schema **and** authorizes every column reference — projection, aggregation args, filters, group_by, order_by, time_range — against the role's column allowlist (the [#223](https://github.com/Wave-RF/WaveHouse/issues/223) hard cap). A full-row read is requested with `select_all`, which expands to the role's allowed columns rather than emitting a raw `SELECT *`; an omitted projection selects nothing, and `*` in `columns` is a literal column name. Every identifier is backtick-quoted via `internal/chsql` (`QuoteIdent`) so any ClickHouse-legal name is accepted — a name containing `?` is refused fail-closed ([#279](https://github.com/Wave-RF/WaveHouse/issues/279)). `InjectPermissionFilters()` adds row-level security. `ApplyMaxRows()` enforces limits. Timestamp bucketing for cache optimization. ### `chsql/` — ClickHouse SQL Helpers @@ -150,13 +150,14 @@ The package's design invariants — stdout always 100%, WARN+ERROR always export ### Ingest Path -```text +```text wrap=false Client POST /v1/ingest?table={table} → JWT auth middleware (always runs; token optional) → Look up table schema from SchemaRegistry → Policy check: role allowed to insert into this table (before the body is parsed) → Validate JSON body against schema (type checks, required columns) - → Policy column rules + check clauses (disallowed columns rejected; claim-derived values enforced or injected) + → Policy column rules + check clauses (disallowed columns rejected; + claim-derived values enforced or injected) → Optional deduplication check (configurable ID field) → Publish to NATS JetStream (ingest.{table}) → 200 OK returned immediately diff --git a/docs/src/content/docs/claude-code.md b/docs/src/content/docs/claude-code.md index 68c61c3e..8d9b71b8 100644 --- a/docs/src/content/docs/claude-code.md +++ b/docs/src/content/docs/claude-code.md @@ -146,7 +146,7 @@ User-specific worktrunk config goes in `~/.config/worktrunk/config.toml`; the co The GitHub MCP server is useful for cross-repo code search and bulk graph queries, but neither is a daily WaveHouse pattern. If you want it, add at user level: -```jsonc +```jsonc title="~/.claude.json" // ~/.claude.json — your user-level config { "mcpServers": { @@ -198,7 +198,7 @@ The deny list blocks: Not committed at project level. Personal preference — put in `.claude/settings.local.json`: -```jsonc +```jsonc title=".claude/settings.local.json" { "statusLine": { "type": "command", "command": "~/.config/claude/statusline.sh" }, "outputStyle": "default", diff --git a/docs/src/content/docs/configuration.md b/docs/src/content/docs/configuration.mdx similarity index 88% rename from docs/src/content/docs/configuration.md rename to docs/src/content/docs/configuration.mdx index ab31eb1e..d01d01cd 100644 --- a/docs/src/content/docs/configuration.md +++ b/docs/src/content/docs/configuration.mdx @@ -5,6 +5,8 @@ sidebar: order: 9 --- +import { Tabs, TabItem } from "@astrojs/starlight/components"; + WaveHouse is configured via a YAML file with environment variable overrides. All environment variables use the `WH_` prefix. ## Loading Order @@ -24,13 +26,13 @@ export WH_CONFIG=/etc/wavehouse/config.yaml ### State | YAML Key | Env Var | Default | Description | -| -------- | ------- | ------- | ----------- | +| --- | --- | ------- | ----------- | | `data_dir` | `WH_DATA_DIR` | `./data` | Root directory for embedded state. NATS JetStream lives at `/nats`; Pebble (when dedupe is enabled) at `/pebble`. Subdirectory names are conventions, not config — one knob, one mount. **In a container this MUST resolve to a host-backed volume**; the relative default is for local binary use. WaveHouse logs a startup `WARN` when the directory is missing or empty (no prior state). See [Persistent Storage](/deployment#persistent-storage-required-for-containers). | ### Server | YAML Key | Env Var | Default | Description | -| -------- | ------- | ------- | ----------- | +| --- | --- | ------- | ----------- | | `server.port` | `WH_SERVER_PORT` | `8080` | HTTP server listen port. | | `server.shutdown_timeout` | `WH_SERVER_SHUTDOWN_TIMEOUT` | `10` | Graceful shutdown timeout in seconds. | | `server.cors_allowed_origins` | `WH_SERVER_CORS_ALLOWED_ORIGINS` | `*` | Comma-separated list of allowed CORS origins. `*` allows any browser origin. WaveHouse is a Bearer-token API — `Access-Control-Allow-Credentials` is intentionally never sent, so this allowlist controls *which origins can read responses*, not cookie scope. Tighten to your frontend's exact origin(s) in production (e.g. `https://dashboard.example.com,http://localhost:3000`). | @@ -38,7 +40,7 @@ export WH_CONFIG=/etc/wavehouse/config.yaml ### ClickHouse | YAML Key | Env Var | Default | Description | -| -------- | ------- | ------- | ----------- | +| --- | --- | ------- | ----------- | | `clickhouse.addr` | `WH_CH_ADDR` | `localhost:9000` | ClickHouse native protocol address. | | `clickhouse.http_port` | `WH_CH_HTTP_PORT` | `8123` | ClickHouse HTTP interface port. Used by the ingest worker (`internal/ingest`) for bulk INSERT and by the raw-SQL proxy (`POST /v1/admin/query`, `internal/api/query.go`) to forward SQL to ClickHouse. Schema discovery uses the native protocol on `addr` instead. | | `clickhouse.http_scheme` | `WH_CH_HTTP_SCHEME` | `http` | HTTP scheme for the ClickHouse HTTP interface (`http` or `https`). Set to `https` for TLS-encrypted ClickHouse connections. | @@ -50,34 +52,34 @@ export WH_CONFIG=/etc/wavehouse/config.yaml ### Schema Discovery | YAML Key | Env Var | Default | Description | -| -------- | ------- | ------- | ----------- | +| --- | --- | ------- | ----------- | | `schema.refresh_interval` | `WH_SCHEMA_REFRESH_INTERVAL` | `60` | How often (in seconds) to re-discover ClickHouse table schemas. Also refreshable on-demand via `POST /v1/schema/refresh` (admin-only). | ### Message Queue (NATS) | YAML Key | Env Var | Default | Description | -| -------- | ------- | ------- | ----------- | +| --- | --- | ------- | ----------- | | `mq.gap_window_minutes` | `WH_MQ_GAP_WINDOW_MINUTES` | `15` | How many minutes of messages to retain in NATS for SSE gap-fill. The Active Sweeper will not purge messages newer than this window. | | `mq.max_bytes_gb` | `WH_MQ_MAX_BYTES_GB` | `50` | Maximum NATS JetStream stream size in GB. When full, new publishes are rejected with `DiscardNew` policy, triggering 503 backpressure on the ingest endpoint. | ### Deduplication | YAML Key | Env Var | Default | Description | -| -------- | ------- | ------- | ----------- | +| --- | --- | ------- | ----------- | | `dedupe.enabled` | `WH_DEDUPE_ENABLED` | `false` | Enable event deduplication. When enabled, the ingest handler checks for duplicates using the configured ID field. | | `dedupe.id_field` | `WH_DEDUPE_ID_FIELD` | `event_id` | JSON field name in the ingest body used as the dedup key. | ### Cache | YAML Key | Env Var | Default | Description | -| -------- | ------- | ------- | ----------- | +| --- | --- | ------- | ----------- | | `cache.l1_max_cost` | `WH_CACHE_L1_MAX_COST` | `67108864` | Maximum L1 cache size in bytes (~64 MB). | | `cache.timestamp_bucket_seconds` | `WH_CACHE_TIMESTAMP_BUCKET_SECONDS` | `60` | Bucket size (seconds) for time-range truncation in structured queries. Improves cache hit rate by normalizing timestamps. | ### Authentication | YAML Key | Env Var | Default | Description | -| -------- | ------- | ------- | ----------- | +| --- | --- | ------- | ----------- | | `auth.jwt_secret` | `WH_AUTH_JWT_SECRET` | *(empty)* | HMAC secret for JWT validation. Set this (or `jwks_url`) so presented tokens are verified; see [Access Control](/access-control). | | `auth.jwks_url` | `WH_AUTH_JWKS_URL` | *(empty)* | JWKS endpoint URL for public key validation (e.g., `https://auth.example.com/.well-known/jwks.json`). When set, JWKS is the **sole** verifier and `jwt_secret` is ignored (not a per-token fallback); the endpoint must be reachable at startup or the server fails to boot. | | `auth.role_claim` | `WH_AUTH_ROLE_CLAIM` | `role` | Dot-separated JWT claim path for role extraction (e.g., `app_metadata.role`). | @@ -94,19 +96,19 @@ See [API — Authentication](/api#authentication). ### Dead Letter Queue (DLQ) | YAML Key | Env Var | Default | Description | -| -------- | ------- | ------- | ----------- | +| --- | --- | ------- | ----------- | | `dlq.enabled` | `WH_DLQ_ENABLED` | `true` | Enable the Dead Letter Queue. Failed batch inserts are published to the `WAVEHOUSE_DLQ` NATS stream instead of blocking retries. | ### Access Control Policy | YAML Key | Env Var | Default | Description | -| -------- | ------- | ------- | ----------- | +| --- | --- | ------- | ----------- | | `policy.file_path` | `WH_POLICY_FILE_PATH` | *(empty)* | Optional path to a YAML/JSON policy file used to seed the policy store on first startup (when NATS KV is empty). **When set, the file MUST exist and parse — WaveHouse refuses to boot otherwise**, so a typo or missing mount surfaces immediately instead of silently denying every request (`Evaluate` fails closed on a `nil` policy, including the admin role). Empty default — no implicit `policy.yaml` lookup — so operators opt into the bootstrap file explicitly; without one, seed the policy via `PUT /v1/admin/policy`. Once KV is populated, the file is ignored on subsequent boots (KV is the source of truth; runtime updates flow through the API and KV Watch). | ### Named Pipes | YAML Key | Env Var | Default | Description | -| -------- | ------- | ------- | ----------- | +| --- | --- | ------- | ----------- | | `pipes.dir` | `WH_PIPES_DIR` | *(empty)* | Optional bootstrap source for named query pipes. When set, `.sql` files in this directory are loaded into the NATS KV pipe store on startup. The directory is **read-only at runtime** — it's a seed, not authoritative storage. After bootstrap, the API and KV are the source of truth (runtime pipe edits go through the API, not the files). Empty default skips bootstrap entirely. Mount read-only in containers (e.g. `./my-pipes:/app/pipes:ro`). | ### OTel @@ -115,10 +117,10 @@ The master switch is `otel.enabled`. When `true`, each signal (traces/metrics/lo **Sampling rates apply only to the OTLP push path.** Stdout always emits 100% of records — operators using a scraping-style pipeline (Promtail/Grafana Alloy → Loki, Vector, Fluent Bit, etc.) set the collection rate at the scraper, not the application. WaveHouse pushes telemetry to an OTel collector; the scraper world owns its own ingest policy. If you want to throttle OTLP volume for cost, lower the rates below. If you want to throttle Loki/Datadog Logs/etc., do it at that pipeline. -**TLS / direct-to-cloud limitation.** The OTLP exporters currently use `WithInsecure()` — plaintext gRPC only. WaveHouse cannot ship directly to TLS-protected OTLP endpoints (Grafana Cloud's OTLP gateway, Honeycomb, Datadog OTLP, etc.). The standard workaround is a sidecar collector (the OTel collector or Grafana Alloy) on `127.0.0.1:4317` that receives our plaintext OTLP and re-exports to the cloud endpoint with TLS + auth headers configured locally. Tracked in #97. +**TLS / direct-to-cloud limitation.** The OTLP exporters currently use `WithInsecure()` — plaintext gRPC only. WaveHouse cannot ship directly to TLS-protected OTLP endpoints (Grafana Cloud's OTLP gateway, Honeycomb, Datadog OTLP, etc.). The standard workaround is a sidecar collector (the OTel collector or Grafana Alloy) on `127.0.0.1:4317` that receives our plaintext OTLP and re-exports to the cloud endpoint with TLS + auth headers configured locally. Tracked in [#97](https://github.com/Wave-RF/WaveHouse/issues/97). | YAML Key | Env Var | Default | Description | -| -------- | ------- | ------- | ----------- | +| --- | --- | ------- | ----------- | | `otel.enabled` | `WH_OTEL_ENABLED` | `false` | Master switch. When `false`, no signals are initialized regardless of the sub-toggles below. | | `otel.addr` | `WH_OTEL_ADDR` | `127.0.0.1:4317` | OTLP gRPC endpoint used by every enabled signal. Plain `host:port` — no scheme, plaintext gRPC only (see TLS note above). See `make help` for a local collector setup. | | `otel.traces.enabled` | `WH_OTEL_TRACES_ENABLED` | `true` | Export traces via OTLP gRPC. | @@ -132,7 +134,7 @@ The master switch is `otel.enabled`. When `true`, each signal (traces/metrics/lo Prometheus exposition is its own top-level config block, independent of `otel.*`. Operators using a scrape-based pipeline (Grafana Alloy, Mimir, the standalone Prometheus server) can leave the entire `[otel]` block at its `enabled: false` default and turn on only `prometheus.enabled` — no OTLP collector required. Conversely, OTLP push and Prometheus can both be on at once (the underlying OTel MeterProvider drives both readers, same `Meter()` API). Disabled by default since enabling adds an unauthenticated endpoint, so opt-in is explicit. | YAML Key | Env Var | Default | Description | -| -------- | ------- | ------- | ----------- | +| --- | --- | ------- | ----------- | | `prometheus.enabled` | `WH_PROMETHEUS_ENABLED` | `false` | Expose a Prometheus-format `/metrics` endpoint. Works standalone (no OTel push) or alongside `otel.metrics.enabled`. | | `prometheus.path` | `WH_PROMETHEUS_PATH` | `/metrics` | URL path. Must start with `/`, and may not collide with a reserved probe path (`/livez`, `/readyz`, `/healthz`, `/health`, `/ready`). When `port` is `0` (mounted on the main server) it also may not be `/v1` or sit under `/v1/`, which would shadow the authenticated API. An invalid path fails validation at startup. | | `prometheus.port` | `WH_PROMETHEUS_PORT` | `0` | Listener port. `0` mounts the endpoint on the main API server (`server.port`) — simplest, no extra port to expose. Non-zero spins up a dedicated HTTP listener, which lets you firewall metrics off the public API surface (common production posture). Must not equal `server.port` when non-zero. | @@ -145,6 +147,10 @@ Prometheus exposition is its own top-level config block, independent of `otel.*` ## Example Config File +Every key, with its default. Save the YAML as `config.yaml` next to the binary (or point `WH_CONFIG` at it), or supply the same settings as environment variables — e.g. an `env_file` in Compose: + + + ```yaml data_dir: ./data # nats → ./data/nats, pebble → ./data/pebble @@ -187,7 +193,8 @@ dlq: enabled: true policy: - file_path: "" # empty = skip bootstrap (seed via PUT /v1/admin/policy); set to a path and the file MUST exist or boot fails + file_path: "" # empty = skip bootstrap (seed via PUT /v1/admin/policy); + # set to a path and the file MUST exist or boot fails pipes: dir: "" # empty = skip bootstrap; set + read-only mount to seed pipes @@ -209,3 +216,55 @@ prometheus: path: /metrics port: 0 # 0 = mount on server.port; non-zero = sidecar listener ``` + + +```ini +WH_DATA_DIR=./data + +WH_SERVER_PORT=8080 +WH_SERVER_SHUTDOWN_TIMEOUT=10 +WH_SERVER_CORS_ALLOWED_ORIGINS=* + +WH_CH_ADDR=localhost:9000 +WH_CH_HTTP_PORT=8123 +WH_CH_HTTP_SCHEME=http +WH_CH_DATABASE=default +WH_CH_USERNAME=default +WH_CH_PASSWORD= +WH_CH_QUERY_TIMEOUT=30s + +WH_MQ_GAP_WINDOW_MINUTES=15 +WH_MQ_MAX_BYTES_GB=50 + +WH_DEDUPE_ENABLED=false +WH_DEDUPE_ID_FIELD=event_id + +WH_CACHE_L1_MAX_COST=67108864 +WH_CACHE_TIMESTAMP_BUCKET_SECONDS=60 + +WH_AUTH_JWT_SECRET=change-me-in-production +WH_AUTH_JWKS_URL= +WH_AUTH_ROLE_CLAIM=role + +WH_SCHEMA_REFRESH_INTERVAL=60 + +WH_DLQ_ENABLED=true + +WH_POLICY_FILE_PATH= + +WH_PIPES_DIR= + +WH_OTEL_ENABLED=false +WH_OTEL_ADDR=127.0.0.1:4317 +WH_OTEL_TRACES_ENABLED=true +WH_OTEL_TRACES_SAMPLE_RATE=1.0 +WH_OTEL_METRICS_ENABLED=true +WH_OTEL_LOGS_ENABLED=true +WH_OTEL_LOGS_SAMPLE_RATE=1.0 + +WH_PROMETHEUS_ENABLED=false +WH_PROMETHEUS_PATH=/metrics +WH_PROMETHEUS_PORT=0 +``` + + diff --git a/docs/src/content/docs/deployment.md b/docs/src/content/docs/deployment.md index 93122d69..eceb0af5 100644 --- a/docs/src/content/docs/deployment.md +++ b/docs/src/content/docs/deployment.md @@ -29,8 +29,10 @@ docker compose -f deployments/compose/standalone.yaml exec clickhouse \ ORDER BY (page) " -# Ingest data (the standalone stack ships a permissive trial policy; WaveHouse is fail-closed otherwise — see Getting Started) -# A 404 "unknown table" right after creating the table means schema discovery hasn't picked it up yet — retry (worst case 60s) +# Ingest data (the standalone stack ships a permissive trial policy; +# WaveHouse is fail-closed otherwise — see Getting Started) +# A 404 "unknown table" right after creating the table means schema +# discovery hasn't picked it up yet — retry (worst case 60s) curl -X POST "http://localhost:8080/v1/ingest?table=clicks" \ -H "Content-Type: application/json" \ -d '{"page": "/home", "button": "signup", "score": 42.5}' @@ -116,7 +118,8 @@ Key variables for production: ```bash # Required WH_CH_ADDR=clickhouse:9000 -WH_CH_HTTP_PORT=8123 # Port for HTTP inserts + /v1/admin/query proxy (default: 8123) +# Port for HTTP inserts + /v1/admin/query proxy (default: 8123) +WH_CH_HTTP_PORT=8123 WH_CH_HTTP_SCHEME=http # Scheme for the same (http/https) # Schema discovery @@ -150,7 +153,8 @@ WH_DEDUPE_ID_FIELD=event_id # Standalone tuning WH_MQ_GAP_WINDOW_MINUTES=15 # Minutes of NATS history for SSE gap-fill -WH_MQ_MAX_BYTES_GB=50 # Max NATS JetStream disk usage (triggers backpressure) +# Max NATS JetStream disk usage (triggers backpressure) +WH_MQ_MAX_BYTES_GB=50 # DLQ WH_DLQ_ENABLED=true # Dead Letter Queue for failed inserts @@ -169,8 +173,10 @@ If `data_dir` resolves into the container's writable overlay layer instead, **Je WaveHouse runs a simple existence check on startup and logs a `WARN` if `/nats` (or `/pebble` when dedupe is on) is missing or empty: -```text -WARN data directory does not exist — starting with no prior state. If this is a redeploy, your persistent volume is not actually persisting; verify your mount. +```text wrap=false +WARN data directory does not exist — starting with no prior state. + If this is a redeploy, your persistent volume is not actually + persisting; verify your mount. ``` On a first-ever run this is expected. On every subsequent run it should be silent — so when this warning *does* fire after a redeploy, that's the most direct signal that the persistent volume isn't actually persisting. @@ -197,8 +203,10 @@ volumes: Bind mounts do **not** copy-up — Docker exposes the host directory as-is, and the image's pre-created dir is masked entirely. If `/srv/wavehouse` is owned by `root:root` on the host (the default for a freshly `mkdir`'d directory), the binary fails at startup with a permission error from NATS: -```text -ERROR mq init failed error="..." path=/app/data/nats hint="if running in a container with a host bind mount, the host directory must be owned by UID 65532..." +```text wrap=false +ERROR mq init failed error="..." path=/app/data/nats + hint="if running in a container with a host bind mount, the host + directory must be owned by UID 65532..." ``` The fix is one host-side command before first start: @@ -306,7 +314,8 @@ K8s `livenessProbe` and `readinessProbe` use kubelet HTTP probes from outside th ```yaml startupProbe: httpGet: { path: /livez, port: 8080 } - failureThreshold: 30 # allow up to 5 min for first schema discovery (30 × periodSeconds) + # allow up to 5 min for first schema discovery (30 × periodSeconds) + failureThreshold: 30 periodSeconds: 10 livenessProbe: httpGet: { path: /livez, port: 8080 } @@ -373,7 +382,8 @@ The underlying Docker run scripts live in `scripts/otel/` and are invoked via Ma ```bash make obs-aspire # Simplest, in-memory, no login make obs-grafana # Full Grafana LGTM stack, auto-login enabled -make obs-front # Simple OTeL Frontend like aspire, with more control over dashboards +# Simple OTeL Frontend like aspire, with more control over dashboards +make obs-front ``` All options automatically listen on standard OTLP ports (`4317` gRPC / `4318` HTTP). If you are running WaveHouse directly on your host (e.g. `make dev`), the default environment variable `WH_OTEL_ADDR=127.0.0.1:4317` will route telemetry to these containers automatically. @@ -412,7 +422,9 @@ docker compose -f deployments/compose/standalone.yaml up -d ### Option 3: Reset for Local Binary Development ```bash -rm -rf data/ # Removes embedded NATS + Pebble data (run `make clean-all` to also drop docker volumes) -make clean # Removes build artifacts: bin/, dist/, clients/ts/dist/, docs/dist/, docs/.dev-dist/ +rm -rf data/ # Removes embedded NATS + Pebble data + # (run `make clean-all` to also drop docker volumes) +make clean # Removes build artifacts: + # bin/, dist/, clients/ts/dist/, docs/dist/, docs/.dev-dist/ make build && ./bin/wavehouse ``` diff --git a/docs/src/content/docs/development.md b/docs/src/content/docs/development.md index ea13f006..c3961a0d 100644 --- a/docs/src/content/docs/development.md +++ b/docs/src/content/docs/development.md @@ -285,18 +285,33 @@ All tests run with Go's **race detector** (`-race`) enabled by default. WaveHous ### Quick Reference ```bash -make test # Unit tests (compact output) — alias for `test-unit` -V=1 make test # Unit tests (verbose output) -make test ARGS="-run TestValidate" # Run specific test(s) -V=1 make test ARGS="-run TestValidate" # Specific test, verbose -make test-integration # Go integration tests (requires Docker) -V=1 make test-integration # Integration tests, verbose -make test-ts # SDK vitest unit tests + coverage + gate against suites.ts-unit - # (`make cov` auto-merges ts-unit + ts-e2e — no separate command) -make test-e2e # E2E SDK suite against bin/wavehouse-cov -make test-all # All four suites sequentially + merged coverage -make ci # Full CI: parallel verify + builds (Go + SDK + docs) + test + test-ts, then test-integration + test-e2e + cov -make cov # Merge available covdata + gate against total threshold +# Prefix any test target with V=1 for verbose output, e.g. `V=1 make test` + +# Unit tests (compact output) — alias for `test-unit` +make test + +# Run specific test(s) +make test ARGS="-run TestValidate" + +# Go integration tests (requires Docker) +make test-integration + +# SDK vitest unit tests + coverage + gate against suites.ts-unit +# (`make cov` auto-merges ts-unit + ts-e2e — no separate command) +make test-ts + +# E2E SDK suite against bin/wavehouse-cov +make test-e2e + +# All four suites sequentially + merged coverage +make test-all + +# Full CI: parallel verify + builds (Go + SDK + docs) + test + test-ts, +# then test-integration + test-e2e + cov +make ci + +# Merge available covdata + gate against total threshold +make cov ``` Each test target writes `covdata` to `tmp/coverage//data/`, renders a textfmt + HTML report, and gates against the per-suite threshold in `.testcoverage.yml`. `make cov` merges whichever suites have run and gates against the total. @@ -341,8 +356,8 @@ The primary E2E integration test suite lives in `tests/e2e/sdk/`. It uses the Ty **Running E2E tests**: ```bash -make test-e2e # Build the cover binary, install deps, run all E2E tests -KEEP_RUNNING=true make test-e2e # Don't tear down services after tests +# Build the cover binary, install deps, run all E2E tests +make test-e2e ``` `make test-e2e` builds `bin/wavehouse-cov` (coverage-instrumented) and runs the orchestrator under `scripts/orchestrator/` to wire ClickHouse + the cover binary into the suite. covdata flushes on SIGINT into `tmp/coverage/e2e/data/`. @@ -532,7 +547,9 @@ git tag sdk-v0.1.0 git push origin sdk-v0.1.0 ``` -> **The first tagged release promotes `latest`.** npm sets a package's `latest` dist-tag on its *first* publish even under `--tag dev`, so until the first `sdk-v*` release a bare `npm install @wavehouse/sdk` (and the bare CDN URLs) resolve to a `0.0.0-dev.*` snapshot. The first tagged stable release moves `latest` to a real version and fixes this for every consumer. +:::caution[The first tagged release promotes `latest`] +npm sets a package's `latest` dist-tag on its *first* publish even under `--tag dev`, so until the first `sdk-v*` release a bare `npm install @wavehouse/sdk` (and the bare CDN URLs) resolve to a `0.0.0-dev.*` snapshot. The first tagged stable release moves `latest` to a real version and fixes this for every consumer. +::: ## CI & review automation diff --git a/docs/src/content/docs/getting-started.md b/docs/src/content/docs/getting-started.md index 2a35101c..5ae0e978 100644 --- a/docs/src/content/docs/getting-started.md +++ b/docs/src/content/docs/getting-started.md @@ -75,7 +75,9 @@ curl -s -X POST "http://localhost:8080/v1/query?table=clicks" \ `POST /v1/query?table={table}` and `GET/POST /v1/pipes/{name}` are cached in-process (L1 Ristretto) with singleflight coalescing — duplicate concurrent queries hit ClickHouse once. For raw SQL there's `POST /v1/admin/query` (an admin escape hatch that never caches, emitting `Cache-Control: no-store`), but it's **admin-only** — the trial `public` role can't reach it. To use it, swap the public default for real auth: configure a JWT secret and present a token whose role is the policy [`admin_role`](/access-control#admin_role--the-privileged-role). -Prefer a type-safe query builder? See the [structured query endpoint](/api#post-v1querytabletable--structured-query) or the [TypeScript SDK](/sdk). +:::tip[Prefer a type-safe client?] +The [TypeScript SDK](/sdk) wraps this endpoint in a chainable query builder with autocomplete on your table names and row types — plus live queries and streaming. The raw shapes are in the [structured query reference](/api#post-v1querytabletable--structured-query). +::: ## 5. Subscribe to real-time updates @@ -89,6 +91,16 @@ curl -N "http://localhost:8080/v1/stream?table=clicks" curl -N "http://localhost:8080/v1/stream?table=clicks&since=2026-03-24T11:00:00Z" ``` +## Troubleshooting first runs + +The handful of things that most often trip up a first session — each is expected behavior with a quick fix: + +- **`404 unknown table: clicks` on the first ingest.** Schema discovery refreshes every 60 seconds (`WH_SCHEMA_REFRESH_INTERVAL`), so a just-created table may not be visible yet. Wait and retry — worst case the next refresh is a full 60 seconds out. (`POST /v1/schema/refresh` forces it, but that endpoint is admin-only — the trial `public` role can't call it.) +- **The query returns `[]` right after an ingest succeeded.** Ingest acknowledges as soon as the event is durable in the WAL; the batch worker flushes to ClickHouse every few seconds. If you query within that window the rows simply aren't in ClickHouse yet — re-query after ~5 seconds. (The [SSE stream](#5-subscribe-to-real-time-updates) sees events *immediately* — it's broadcast before the flush.) +- **`403` on a table you created yourself.** WaveHouse is fail-closed and the trial policy grants the `public` role access to the *named demo tables only* (`clicks`, `events`). A new table needs a policy entry — see [Access Control](/access-control) for granting roles per table. +- **A port is already taken.** The stack binds `8080` (WaveHouse) and `8123`/`9000` (ClickHouse). Stop whatever holds the port or edit the `ports:` mappings in `deployments/compose/standalone.yaml`. +- **Errors right at first boot.** `docker compose -f deployments/compose/standalone.yaml ps` should show both services up — ClickHouse takes a few seconds to initialize on a cold start, so give the stack a moment before the first request. + ## Next steps - **[Architecture](/architecture)** — how ingest, query, cache, and streaming fit together. diff --git a/docs/src/content/docs/index.mdx b/docs/src/content/docs/index.mdx index 6cefa59a..b10e97b1 100644 --- a/docs/src/content/docs/index.mdx +++ b/docs/src/content/docs/index.mdx @@ -25,7 +25,7 @@ hero: variant: minimal --- -import { Card, CardGrid, LinkCard, LinkButton } from '@astrojs/starlight/components'; +import { Card, CardGrid, LinkCard, LinkButton, Tabs, TabItem } from '@astrojs/starlight/components'; import HomepageCtaTracking from '../../components/HomepageCtaTracking.astro';
@@ -102,6 +102,50 @@ If you're building user-facing analytics, **WaveHouse is like Supabase for Click **Plus** — optional [deduplication](/configuration#deduplication) (idempotent ingest by ID), a dead-letter queue for failed batch inserts, and Tinybird-style [named pipes](/pipes) with parameter binding and per-role restrictions. +## Query it like a database. Subscribe to it like a socket. + +The zero-dependency [TypeScript SDK](/sdk) wraps the whole surface — typed inserts, a chainable query builder, and live queries that backfill history before streaming: + + + + ```ts + import { createClient } from '@wavehouse/sdk'; + + const wh = createClient({ baseURL: 'https://wavehouse.example.com' }); + + // Returns immediately — buffered in the WAL, schema-validated + // at the edge, batch-flushed to ClickHouse behind the scenes. + await wh.from('clicks').insert({ page: '/home', button: 'signup' }); + ``` + + + ```ts + // Chainable, type-safe, never throws — branch on { data, error }. + const { data, error } = await wh.from('clicks') + .select('page', 'button', 'score') + .where('page', '=', '/home') + .orderBy('received_timestamp', 'desc') + .limit(100); + ``` + + + ```ts + // Historical backfill + real-time stream, deduplicated at the seam. + const lq = wh.from('clicks') + .selectAll() + .where('page', '=', '/home') + .orderBy('received_timestamp', 'desc') + .limit(100) + .liveQuery({ + initial: (result) => setRows(result.data ?? []), + next: (event) => addRow(event.data), + }); + + // later: lq.close(); + ``` + + + ## Up in five minutes One compose file — ClickHouse + WaveHouse — and you have an ingest endpoint, a query endpoint, and a live event stream: @@ -123,11 +167,7 @@ curl -N "http://localhost:8080/v1/stream?table=clicks" WaveHouse is fail-closed, so the standalone stack ships a permissive trial policy (a non-admin `public` role that can read/write the demo tables, seeded on first boot) — then it's five minutes from `git clone` to a live event stream. See [Getting Started](/getting-started). -## How it ships - - - One `wavehouse` process — API, batch worker, embedded NATS JetStream, and optional embedded Pebble dedup, all in-process. **The only external dependency is ClickHouse.** Same binary on a laptop, in Docker, or on a production host. - +**How it ships:** one `wavehouse` process — API, batch worker, embedded NATS JetStream, and optional embedded Pebble dedup, all in-process. **The only external dependency is ClickHouse.** Same binary on a laptop, in Docker, or on a production host. ## Where to next diff --git a/docs/src/content/docs/ingest-pipeline.md b/docs/src/content/docs/ingest-pipeline.md index 24f0b367..ac2809ac 100644 --- a/docs/src/content/docs/ingest-pipeline.md +++ b/docs/src/content/docs/ingest-pipeline.md @@ -135,15 +135,15 @@ the moment the slot frees. ```mermaid stateDiagram-v2 [*] --> Idle - Idle --> Accumulating: add first row (arm timer) - Accumulating --> Accumulating: add (batch under maxBatch) - Accumulating --> Flushing: size hit or timer fires (start flush) - Flushing --> Flushing: add (accumulate a fresh batch) - Flushing --> FlushingQueued: size hit or timer fires (latch flushQueued) + Idle --> Accumulating: first row (arm timer) + Accumulating --> Accumulating: add (under maxBatch) + Accumulating --> Flushing: size or timer — flush + Flushing --> Flushing: add (fresh batch) + Flushing --> FlushingQueued: size or timer — latch FlushingQueued --> FlushingQueued: add - Flushing --> Accumulating: flush done, rows remain (wait for own trigger) - Flushing --> Idle: flush done, batch empty - FlushingQueued --> Flushing: flush done, start the deferred flush + Flushing --> Accumulating: done — rows remain + Flushing --> Idle: done — empty + FlushingQueued --> Flushing: done — deferred flush ``` Two consequences worth internalizing: diff --git a/docs/src/content/docs/pipes.md b/docs/src/content/docs/pipes.mdx similarity index 90% rename from docs/src/content/docs/pipes.md rename to docs/src/content/docs/pipes.mdx index de58f264..f640414f 100644 --- a/docs/src/content/docs/pipes.md +++ b/docs/src/content/docs/pipes.mdx @@ -5,6 +5,8 @@ sidebar: order: 6 --- +import { Tabs, TabItem } from "@astrojs/starlight/components"; + A **named pipe** is a saved SQL query, registered under a name, that callers run by name with parameters — without ever sending raw SQL. They turn an ad-hoc query into a stable, cached, access-controlled endpoint: you write the SQL once as an operator, expose it at `GET/POST /v1/pipes/{name}`, and clients supply only the declared parameters. Pipes are the right tool when a query is reusable and shouldn't live in client code — dashboards, reports, public APIs over curated slices of data. They sit on the **cached read path** (shared L1 + singleflight, same as structured queries), and authorize through a simple per-pipe allowlist rather than the full [policy engine](/access-control). @@ -33,7 +35,9 @@ A pipe is defined by five fields: } ``` -> **Pipe SQL is operator-authored and admin-only to write.** Anyone who can create or edit a pipe can run arbitrary SQL, so pipe management lives entirely behind the admin gate. Callers of a pipe never send SQL — only parameter values. Keep that boundary in mind when you design a pipe: the SQL is your trusted code, the parameters are untrusted input. +:::caution[Pipe SQL is operator-authored and admin-only to write] +Anyone who can create or edit a pipe can run arbitrary SQL, so pipe management lives entirely behind the admin gate. Callers of a pipe never send SQL — only parameter values. Keep that boundary in mind when you design a pipe: the SQL is your trusted code, the parameters are untrusted input. +::: ## Parameters @@ -157,20 +161,30 @@ The response is a JSON array of rows. Results flow through the shared in-process ## Bootstrapping from `.sql` files -For version-controlled, deploy-time pipe definitions, point `pipes.dir` at a directory of `.sql` files: +For version-controlled, deploy-time pipe definitions, point `pipes.dir` (in `config.yaml`, or its environment twin) at a directory of `.sql` files: + + ```yaml -# config.yaml pipes: dir: /app/pipes # seed .sql files into KV on startup ``` + + +```ini +WH_PIPES_DIR=/app/pipes +``` + + On startup, each top-level `*.sql` file is loaded as a pipe whose **name is the filename** (without the extension) and whose **SQL is the file contents**. So `pipes/top_pages.sql` becomes the pipe `top_pages`. The directory is a one-time **seed**, not authoritative storage: - It will **not overwrite** a pipe that already exists in KV — once a pipe is in the store (seeded earlier, or created via the API), the file is skipped. - After bootstrap, the API and KV are the source of truth; the directory is read-only at runtime. Mount it read-only in containers (e.g. `./my-pipes:/app/pipes:ro`). -> **Bootstrapped pipes are admin-only until you grant roles.** A `.sql` file carries only the SQL — there's no way to express `allowed_roles`, `parameters`, or a `description` in the file. A pipe seeded this way therefore has an empty `allowed_roles`, which (fails-closed) means **admin only**. To expose it to other roles or add formal parameters, `PUT` the full definition through `/v1/admin/pipes/{name}` after boot. Inline `{{name:default}}` placeholders still work in a bootstrapped file, so the SQL can be parameter-ready even without formal declarations. +:::caution[Bootstrapped pipes are admin-only until you grant roles] +A `.sql` file carries only the SQL — there's no way to express `allowed_roles`, `parameters`, or a `description` in the file. A pipe seeded this way therefore has an empty `allowed_roles`, which (fails-closed) means **admin only**. To expose it to other roles or add formal parameters, `PUT` the full definition through `/v1/admin/pipes/{name}` after boot. Inline `{{name:default}}` placeholders still work in a bootstrapped file, so the SQL can be parameter-ready even without formal declarations. +::: ## End-to-end example diff --git a/docs/src/content/docs/sdk.md b/docs/src/content/docs/sdk.md deleted file mode 100644 index ce4e5bc2..00000000 --- a/docs/src/content/docs/sdk.md +++ /dev/null @@ -1,799 +0,0 @@ ---- -title: "TypeScript SDK" -description: "Zero-dependency client SDK — query builder, real-time streaming, codegen." -sidebar: - order: 8 ---- - -`@wavehouse/sdk` — Zero-dependency TypeScript client for WaveHouse. - -## Installation - -```bash -npm install @wavehouse/sdk -``` - -This works in any bundler-based framework — React, Vue, Svelte, Angular, Astro, SolidJS, or plain Vite — via `import { createClient } from '@wavehouse/sdk'`. - -### No build step (CDN) - -No bundler or `npm` required — drop the SDK into a static HTML file you deploy over FTP, object storage, or any static host. The ES module loads natively in modern browsers: - -```html - -``` - -Pin a version for production (`https://esm.sh/@wavehouse/sdk@0.1.0`); jsDelivr (`.../+esm`) and unpkg (`?module`) serve the same module. For pages that can't use ES modules, the bundled IIFE build at `https://cdn.jsdelivr.net/npm/@wavehouse/sdk` exposes a `WaveHouse` global (`WaveHouse.createClient({ … })`) for a classic ` + ``` + + Pin a version for production (`https://esm.sh/@wavehouse/sdk@0.1.0`); + jsDelivr (`.../+esm`) and unpkg (`?module`) serve the same module. For + pages that can't use ES modules, the bundled IIFE build at + `https://cdn.jsdelivr.net/npm/@wavehouse/sdk` exposes a `WaveHouse` global + (`WaveHouse.createClient({ … })`) for a classic ` + +``` + +### Runtime support + +**Browsers** — all SDK features work natively. `fetch`, `AbortController`, +and `EventSource` are built into every modern browser; no polyfills are +required. + +**Node.js** — non-streaming features (`wh.from().fetch()`, `.insert()`, +`.sql()`, pipes, admin) work in Node 18 and later (the package's minimum, +per `engines.node`). Streaming (`.stream()`, `.liveQuery()`) uses +`EventSource`, which is **not** a default global in Node. The SDK +feature-detects with `typeof EventSource === "undefined"` and throws a +descriptive error if it is absent. To use streaming in Node you must +polyfill before calling any stream method: + +```js +import { EventSource } from 'eventsource'; // npm install eventsource +globalThis.EventSource = EventSource; +``` + +`--experimental-eventsource` (available in recent Node releases) also +satisfies the check when passed on the command line. + +## Quick Start + + + + ```ts + import { createClient } from '@wavehouse/sdk'; + + const wh = createClient({ + baseURL: 'http://localhost:8080', + auth: async () => getAccessToken(), // omit for public/unauthenticated + }); + + // Query + const { data, error } = await wh.from('clicks').select('page').limit(10); + + // Insert + await wh.from('clicks').insert({ page: '/home', button: 'signup' }); + + // Stream + const stream = wh.from('clicks').stream(); + const unsub = stream.subscribe({ + next: (event) => console.log(event.data), + status: (s) => console.log('Stream:', s), + }); + ``` + + + ```ts + import { createClient } from '@wavehouse/sdk'; + + const wh = createClient({ + baseURL: 'http://localhost:8080', + auth: async () => getAccessToken(), // omit for public/unauthenticated + }); + + // Query + const { data, error } = await wh.from('clicks').select('page').limit(10); + + // Insert + await wh.from('clicks').insert({ page: '/home', button: 'signup' }); + + // Stream + const stream = wh.from('clicks').stream(); + const unsub = stream.subscribe({ + next: (event) => console.log(event.data), + status: (s) => console.log('Stream:', s), + }); + ``` + + + ```ts + import { createClient } from '@wavehouse/sdk'; + + const wh = createClient({ + baseURL: 'http://localhost:8080', + auth: async () => getAccessToken(), // omit for public/unauthenticated + }); + + // Query + const { data, error } = await wh.from('clicks').select('page').limit(10); + + // Insert + await wh.from('clicks').insert({ page: '/home', button: 'signup' }); + + // Stream + const stream = wh.from('clicks').stream(); + const unsub = stream.subscribe({ + next: (event) => console.log(event.data), + status: (s) => console.log('Stream:', s), + }); + ``` + + + ```ts + import { createClient } from '@wavehouse/sdk'; + + const wh = createClient({ + baseURL: 'http://localhost:8080', + auth: async () => getAccessToken(), // omit for public/unauthenticated + }); + + // Query + const { data, error } = await wh.from('clicks').select('page').limit(10); + + // Insert + await wh.from('clicks').insert({ page: '/home', button: 'signup' }); + + // Stream + const stream = wh.from('clicks').stream(); + const unsub = stream.subscribe({ + next: (event) => console.log(event.data), + status: (s) => console.log('Stream:', s), + }); + ``` + + + ```ts + import { createClient } from '@wavehouse/sdk'; + + const wh = createClient({ + baseURL: 'http://localhost:8080', + auth: async () => getAccessToken(), // omit for public/unauthenticated + }); + + // Query + const { data, error } = await wh.from('clicks').select('page').limit(10); + + // Insert + await wh.from('clicks').insert({ page: '/home', button: 'signup' }); + + // Stream + const stream = wh.from('clicks').stream(); + const unsub = stream.subscribe({ + next: (event) => console.log(event.data), + status: (s) => console.log('Stream:', s), + }); + ``` + + + ```ts + import { createClient } from 'https://esm.sh/@wavehouse/sdk'; + + const wh = createClient({ + baseURL: 'http://localhost:8080', + auth: async () => getAccessToken(), // omit for public/unauthenticated + }); + + // Query + const { data, error } = await wh.from('clicks').select('page').limit(10); + + // Insert + await wh.from('clicks').insert({ page: '/home', button: 'signup' }); + + // Stream + const stream = wh.from('clicks').stream(); + const unsub = stream.subscribe({ + next: (event) => console.log(event.data), + status: (s) => console.log('Stream:', s), + }); + ``` + + + +## Creating a Client + +```ts +import { createClient } from '@wavehouse/sdk'; +import type { Database } from './my-types'; // optional hand-written types + +const wh = createClient({ + baseURL: 'https://wavehouse.example.com', + auth: async () => myAuthProvider.getToken(), + options: { + maxRetries: 2, + }, +}); +``` + +### `ClientConfig` + +| Field | Type | Default | Description | +|-------|------|---------|-------------| +| `baseURL` | `string` | — | WaveHouse server URL (required) | +| `auth` | `() => Promise \| string` | — | Token provider. Omit for public access | +| `options.maxRetries` | `number` | `2` | Retry attempts for failed/5xx requests | + +:::note[How the token is transmitted] +The SDK attaches your `auth` token as an `Authorization: Bearer` header on REST calls, and — because the browser `EventSource` API can't set headers — as a `?token=` query parameter on streaming connections. When both are present the server reads the header in preference to the query parameter, and strips the `?token=` value from the URL after extraction so it can't leak into logs. +::: + +### Type-Safe Tables + +Pass a `Database` type to get autocomplete on table names and row types: + +```ts +interface Database { + clicks: { page: string; button: string; score: number; received_timestamp: string }; + users: { id: string; name: string; email: string }; +} + +const wh = createClient({ baseURL: '...' }); +const clicks = wh.from('clicks'); // ✅ autocomplete +const { data } = await clicks.select('page', 'button').limit(10); +// data is Array<{ page: string; button: string; score: number; received_timestamp: string }> | null +``` + +Generate the `Database` interface from a running server with the +[codegen CLI](/sdk/reference#codegen-cli). + +## Result Type + +Every async SDK operation returns `Result` — a discriminated union that never throws: + +```ts +type Result = + | { ok: true; data: T; error: null; hasMore?: boolean; next?: () => Promise> } + | { ok: false; data: null; error: WaveHouseError } + +interface WaveHouseError { + status: number; // HTTP status (0 for network errors) + code: string; // e.g. 'HTTP_400', 'NETWORK_ERROR', 'ABORTED' + message: string; // Human-readable error message + details?: unknown; // Raw response body + retryable: boolean; // Whether SDK would retry this error +} +``` + +Usage pattern — branch on `result.ok` (or destructure `{ data, error }` if you prefer): + +```ts +const result = await wh.from('clicks').select('page').limit(10); +if (result.ok) { + console.log(result.data); // Row[] — TypeScript knows data is non-null here +} else { + console.error(result.error.message); // never throws +} +``` + +The full error-code table lives in +[Error Handling](/sdk/reference#error-handling). + +## Explore the SDK + + + + + + + + diff --git a/docs/src/content/docs/sdk/pipes.md b/docs/src/content/docs/sdk/pipes.md new file mode 100644 index 00000000..3e1cca2f --- /dev/null +++ b/docs/src/content/docs/sdk/pipes.md @@ -0,0 +1,53 @@ +--- +title: "SDK Pipes" +description: "Execute and manage named query pipes with @wavehouse/sdk." +--- + +Named pipes are server-defined, parameterized queries — the +[Named Pipes guide](/pipes) covers defining them. The SDK executes pipes for +any allowed role and manages their definitions under the admin role. +Examples import from `@wavehouse/sdk`; using the CDN instead, import from +`https://esm.sh/@wavehouse/sdk` (see [Imports & Runtimes](/sdk#imports--runtimes)). + +## Named Pipes — `wh.pipe(name, params?)` + +Execute a pre-defined named query pipe. Returns a `PipeRef` which is **PromiseLike**. + +```ts +// These are equivalent (PipeRef is PromiseLike): +const { data } = await wh.pipe('top_pages', { start_date: '2026-01-01', limit: 50 }).fetch(); +const { data } = await wh.pipe('top_pages', { start_date: '2026-01-01', limit: 50 }); +``` + +### `.fetch(opts?)` + +Execute and return results. + +### `.stream(opts?)` + +Open a live stream. See [Streaming](/sdk/streaming). + +--- + +## Pipes Admin — `wh.pipes` + +Manage named query pipes. Requires the admin role (`policy.admin_role`). + +```ts +// List all pipes +const { data: pipes } = await wh.pipes.list(); + +// Get a single pipe definition +const { data: pipe } = await wh.pipes.get('top_pages'); + +// Create or update +await wh.pipes.set('top_pages', { + sql: 'SELECT page, count() as views FROM clicks GROUP BY page LIMIT {{limit}}', + parameters: [{ name: 'limit', type: 'number', required: false, default: 100 }], + description: 'Top pages by view count', + allowed_roles: ['viewer', 'admin'], +}); + +// Delete +await wh.pipes.delete('old_pipe'); +``` diff --git a/docs/src/content/docs/sdk/queries.md b/docs/src/content/docs/sdk/queries.md new file mode 100644 index 00000000..a240693a --- /dev/null +++ b/docs/src/content/docs/sdk/queries.md @@ -0,0 +1,261 @@ +--- +title: "SDK Queries" +description: "Tables, the chainable query builder, pagination, and raw SQL in @wavehouse/sdk." +--- + +Reading and writing data with `@wavehouse/sdk`: table references, the +chainable query builder, cursor pagination, and the admin-only raw-SQL +escape hatch. Every call returns the SDK's +[`Result`](/sdk#result-type) — nothing throws. +Examples import from `@wavehouse/sdk`; using the CDN instead, import from +`https://esm.sh/@wavehouse/sdk` (see [Imports & Runtimes](/sdk#imports--runtimes)). + +## Tables — `wh.from(table)` + +`from()` returns a `TableRef` — a reference to a table. It is **NOT thenable**, so it's safe to pass around or store in a variable without triggering requests. + +```ts +const clicks = wh.from('clicks'); +``` + +### `.fetch(opts?)` + +Shortcut for "select every column", with a default limit of 1000. When an access-control policy restricts your role's columns, the server returns only the columns your role is allowed to read — `.fetch()` is never a way around `deny_columns`/`allow_columns` (see [Access control](/access-control#column-permissions)). + +To paginate, chain an explicit `.orderBy()` — a bare `.fetch()` sends no default order (see [Pagination](#pagination)). Ordering, grouping, or filtering by a column your role can't read is rejected, so a column-restricted role must reference only readable columns in those clauses. + +```ts +const { data, error, hasMore, next } = await clicks.fetch(); +const { data } = await clicks.fetch({ limit: 50, signal: controller.signal }); +``` + +### `.insert(data, opts?)` + +Insert one row or many. A single object is sent as a JSON `POST /v1/ingest?table={table}`. An **array** is serialized to NDJSON (one record per line) and sent as a single `application/x-ndjson` request, so a bad record no longer fails or hides the rest of the batch — per-record outcomes come back in the result. + +```ts +// Single row → { ok: true } (or { ok: true, duplicate: true } when dedup skips it) +const { data, error } = await clicks.insert({ page: '/home', button: 'cta' }); + +// Many rows → one NDJSON request, per-record summary +const { data } = await clicks.insert([ + { page: '/home', button: 'cta' }, + { page: '/about', button: 'nav' }, +]); +// data: { ok, total, succeeded, failed, duplicates, results? } +``` + +For an array insert, `data.ok` is `true` only when every record succeeded (`failed === 0`). Inspect `data.failed` and `data.results` (each `{ index, ok|duplicate|error }`, 1-based `index`) for partial failures — the call's top-level `error` is reserved for whole-request failures (network, `404` unknown table, `403` forbidden, `503` backpressure). An empty array is a no-op and sends no request. The array path sends one request regardless of size; bounded-concurrency chunking of very large arrays is tracked in [#196](https://github.com/Wave-RF/WaveHouse/issues/196). + +> The server itself is format-agnostic: `POST /v1/ingest` also accepts a raw JSON array or a single object directly (the `Content-Type` is only a hint), so non-SDK clients can send whichever shape is convenient. See the [API reference](/api#post-v1ingesttabletable--ingest-data). + +### `.insertNDJSON(source, opts?)` + +Insert pre-formatted NDJSON you already have — a `.ndjson` file, a byte stream, or a string — without first parsing it into objects. Accepts a `string`, `Uint8Array`, `Blob`/`File`, or `ReadableStream`; non-string sources are read fully into memory before sending. Returns the same per-record summary as an array `insert`. + +```ts +// From a string +await clicks.insertNDJSON('{"page":"/a"}\n{"page":"/b"}\n'); + +// From a browser (a File is a Blob) +await clicks.insertNDJSON(fileInput.files[0]); + +// From a Node file (Node 20+: fs.openAsBlob; or read it to a string) +import { openAsBlob } from 'node:fs'; +await clicks.insertNDJSON(await openAsBlob('events.ndjson')); +``` + +### `.schema(opts?)` + +Fetch the table's column definitions from ClickHouse. + +```ts +const { data } = await clicks.schema(); +// data: { name: 'clicks', columns: [ +// { name: 'page', type: 'String', is_nullable: false, has_default: false }, ... +// ] } +``` + +### `.select(...columns)` + +Start a query builder chain. See [Query Builder](#query-builder). + +```ts +const { data } = await clicks.select('page', 'button').where('page', '=', '/home').limit(10); +``` + +### `.selectAll()` + +Start a query that selects **every column your role is allowed to read** — the explicit form of what a bare `.fetch()` does. Mutually exclusive with `.select(...)` and with aggregations (`.count()`, `.sum()`, etc.); the server expands it to your allowed columns (never a raw `SELECT *`) and never bypasses `deny_columns`/`allow_columns`. See [Access control → Column permissions](/access-control#column-permissions). + +```ts +const { data } = await clicks.selectAll().where('country', '=', 'US').limit(10); +``` + +### `.stream(opts?)` + +Open a real-time event subscription. See [Streaming](/sdk/streaming). + +```ts +const stream = clicks.stream({ since: '2026-01-01T00:00:00Z' }); +``` + +--- + +## Query Builder + +Returned by `tableRef.select()`. Immutable — every chain method returns a new `QueryBuilder`. The builder is **PromiseLike**, so `await builder` auto-executes `.fetch()`. + +```ts +// These are equivalent: +const result = await clicks.select('page').limit(10).fetch(); +const result = await clicks.select('page').limit(10); // PromiseLike shortcut +``` + +### Chain Methods + +All methods return a new `QueryBuilder` — the original is unchanged. + +#### `.select(...columns)` + +Append columns to the SELECT clause. A literal `'*'` is the column *named* `*`, not a wildcard — use `.selectAll()` for all columns. + +```ts +const q = clicks.select('page').select('button'); // SELECT page, button +``` + +#### `.selectAll()` + +Select every column your role may read (the all-columns wildcard, expanded server-side to your allowed columns). Mutually exclusive with `.select(...)` and with aggregations (`.count()`, `.sum()`, etc.). + +```ts +const q = clicks.selectAll().where('country', '=', 'US'); +``` + +#### `.where(column, op, value)` + +Add a filter condition. SDK operators are translated to backend format. + +```ts +clicks.select('page').where('score', '>', 10).where('page', 'like', '/home%') +``` + +| SDK Operator | Backend | Description | +|-------------|---------|-------------| +| `'='` | `eq` | Equal | +| `'!='` | `neq` | Not equal | +| `'>'` | `gt` | Greater than | +| `'>='` | `gte` | Greater than or equal | +| `'<'` | `lt` | Less than | +| `'<='` | `lte` | Less than or equal | +| `'in'` | `in` | Value in array | +| `'like'` | `like` | SQL LIKE pattern | +| `'not_like'` | — | SQL NOT LIKE — **client-side only** (live-query / stream filtering); the `/v1/query` backend rejects it | + +#### Aggregations + +```ts +clicks.select('page') + .count('*', 'total') // COUNT(*) + .sum('score', 'total_score') // SUM(score) + .avg('score', 'avg_score') // AVG(score) + .min('score', 'min_score') // MIN(score) + .max('score', 'max_score') // MAX(score) + .countDistinct('page', 'unique_pages') + .aggregate('uniqExact', 'user_id', 'unique_users') // custom fn +``` + +Each aggregation method signature: `(column: string, alias?: string)`. +`count()` defaults to `column='*'`, `alias='count'`. + +#### `.groupBy(...columns)` + +```ts +clicks.select('page').count().groupBy('page') +``` + +#### `.orderBy(column, dir?)` + +```ts +clicks.select('page').count('*', 'total').orderBy('total', 'desc') +``` + +`dir` defaults to `'asc'`. + +#### `.limit(n)` + +```ts +clicks.select().limit(100) +``` + +If no limit is specified, `QueryBuilder.DEFAULT_LIMIT` (1000) is applied automatically to prevent unbounded result sets. The server also enforces a maximum of 10,000 rows (`DefaultMaxRows`). + +#### `.timeRange(column, since, until?)` + +Filter by a time window. `since` and `until` accept RFC3339 timestamps or relative durations (`'1h'`, `'30m'`, `'7d'`, `'2w'` — day and week suffixes expand to hours, so `'7d'` is `'168h'`). + +```ts +clicks.select('page').timeRange('received_timestamp', '1h') +clicks.select('page').timeRange( + 'received_timestamp', '2026-01-01T00:00:00Z', '2026-02-01T00:00:00Z' +) +``` + +#### `.cacheTTL(seconds)` + +Records a desired result-cache TTL on the builder. **Currently client-side state only** — the value is never sent to the server, which derives each result's cache TTL adaptively from query execution time. Wiring it through the wire format is tracked in [#280](https://github.com/Wave-RF/WaveHouse/issues/280). + +```ts +clicks.select('page').count().cacheTTL(300) // not yet honored server-side — see #280 +``` + +### `.fetch(opts?)` + +Execute the query. Returns `Result` with optional pagination. + +```ts +const { data, error, hasMore, next } = await clicks.select('page').limit(50).fetch(); + +if (hasMore && next) { + const page2 = await next(); // cursor-based pagination +} +``` + +**Options:** + +| Field | Type | Description | +|-------|------|-------------| +| `signal` | `AbortSignal` | Cancel the request | +| `limit` | `number` | Override builder limit for this fetch | + +### `.stream(opts?)` + +Open a live stream from the builder's table. See [Streaming](/sdk/streaming). + +### Pagination + +When `limit` is set and the result contains at least `limit` rows, `hasMore` is `true`. Cursor-based pagination's `next()` walks an **order column** — it adds a filter on that column using the last row's value — so `next()` is only attached when the query has an explicit `.orderBy()`. With no order column the result still reports `hasMore` honestly, but `next` is `undefined` (there is no deterministic cursor to build) — add an `.orderBy()` to paginate. + +```ts +let result = await clicks.select().orderBy('received_timestamp', 'desc').limit(100).fetch(); + +const allRows = [...result.data!]; +while (result.hasMore && result.next) { + result = await result.next(); + if (result.data) allRows.push(...result.data); +} +``` + +--- + +## Raw SQL — `wh.sql(query, opts?)` + +Execute a raw SQL query. `/v1/admin/query` is admin-only: the caller's JWT must resolve to the policy admin role (`admin_role`, `"admin"` by default). A request with no token, or an invalid/expired one, falls back to the `default_role` and is rejected. + +```ts +const { data, error } = await wh.sql('SELECT page, count() FROM clicks GROUP BY page LIMIT 10'); +``` + +:::note[No parameter binding through the SDK] +Positional `?` substitution is not supported, and the SDK has no way to forward ClickHouse-style named params (the `WHERE id = {id:UInt32}` + `param_id=42` query-string combo) — the proxy doesn't forward arbitrary query-string params and `wh.sql()` doesn't expose a hook to add them. Inline literals into the SQL, or — for safe binding from user-supplied input — use the structured query builder (`wh.from(table)…`). +::: diff --git a/docs/src/content/docs/sdk/reference.md b/docs/src/content/docs/sdk/reference.md new file mode 100644 index 00000000..f631d1a7 --- /dev/null +++ b/docs/src/content/docs/sdk/reference.md @@ -0,0 +1,154 @@ +--- +title: "SDK Reference & CLI" +description: "Error codes, AbortController, the full API tree, the codegen CLI, and E2E testing with @wavehouse/sdk." +--- + +Cross-cutting reference for `@wavehouse/sdk`: cancellation, the error model +behind every [`Result`](/sdk#result-type), the complete API tree at a +glance, and the tooling that ships in the package. + +## AbortController Support + +All async operations accept an `AbortSignal` for cancellation: + +```ts +const controller = new AbortController(); +setTimeout(() => controller.abort(), 5000); // 5s timeout + +const { data, error } = await wh.from('clicks').fetch({ signal: controller.signal }); +if (error?.code === 'ABORTED') { + console.log('Request timed out'); +} +``` + +--- + +## Error Handling + +The SDK **never throws**. All errors are returned in `Result.error`. + +| Status | Code | Retryable | Description | +|--------|------|-----------|-------------| +| 400 | `HTTP_400` | No | Bad request (validation, missing fields) | +| 401 | `HTTP_401` | No | Missing or invalid JWT | +| 403 | `HTTP_403` | No | Insufficient permissions | +| 404 | `HTTP_404` | No | Table or pipe not found | +| 500 | `HTTP_500` | Yes | Server error (retried per `maxRetries`) | +| 503 | `HTTP_503` | Yes | Service unavailable (auto-retries with `Retry-After`) | +| 0 | `NETWORK_ERROR` | Yes | Network failure (retried with exponential backoff) | +| 0 | `ABORTED` | No | Request canceled via `AbortSignal` | + +--- + +## Full API Tree + +```text +createClient(config) → WaveHouseClient +├── .from(table) → TableRef (NOT thenable) +│ ├── .fetch(opts?) → Promise> +│ ├── .select(...cols?) → QueryBuilder (PromiseLike) +│ │ ├── .select() .selectAll() .where() .count() .sum() .avg() .min() .max() +│ │ │ .countDistinct() .aggregate() .groupBy() .orderBy() +│ │ │ .limit() .timeRange() .cacheTTL() +│ │ ├── .fetch(opts?) → Promise> +│ │ ├── .stream(opts?) → StreamController +│ │ └── .liveQuery(subscriber, opts?) → LiveQuery +│ ├── .selectAll() → QueryBuilder (PromiseLike) +│ ├── .insert(data) → Promise> +│ ├── .insertNDJSON(source) → Promise> +│ ├── .schema() → Promise> +│ └── .stream(opts?) → StreamController +├── .pipe(name, params?) → PipeRef (PromiseLike) +│ ├── .fetch(opts?) → Promise> +│ └── .stream(opts?) → StreamController +├── .pipes (admin) +│ ├── .list() → Promise> +│ ├── .get(name) → Promise> +│ ├── .set(name, def) → Promise> +│ └── .delete(name) → Promise> +├── .sql(query, opts?) → Promise> +├── .schema +│ ├── .list() → Promise> +│ └── .refresh() → Promise> +├── .policy (admin) +│ ├── .get() → Promise> +│ ├── .set(policy) → Promise> +│ └── .validate(policy) → Promise> +├── .dlq +│ ├── .list() → Promise> +│ ├── .table(name) → Promise> +│ └── .stream() → StreamController // not yet functional server-side — #197 +└── .sys + └── .health() → Promise> + +StreamController (NOT thenable) +├── .subscribe({ next, status?, error? }) → unsubscribe() +├── .close() +├── .status → StreamStatus +└── [Symbol.asyncIterator]() → AsyncIterableIterator +``` + +## Codegen CLI + +Generate TypeScript types from a running WaveHouse instance. The package ships a `wavehouse-codegen` bin, so after installing `@wavehouse/sdk` you can run it with `npx`: + +```bash +npx wavehouse-codegen --url http://localhost:8080 --out ./src/db.d.ts + +# Or, working inside this repo (clients/ts/): +pnpm codegen --url http://localhost:8080 --out ./src/db.d.ts +``` + +Codegen reads `/v1/schema`, which is **admin-only**. Against a non-dev server, pass an admin-role token with `--auth ` or the request is denied with `403`. + +**Options:** + +| Flag | Description | Default | +|------|-------------|---------| +| `--url`, `-u` | WaveHouse base URL | `http://localhost:8080` | +| `--out`, `-o` | Output .d.ts file path | `./wavehouse.d.ts` | +| `--auth`, `-a` | Bearer token (if auth required) | — | + +**Example output:** + +```ts +// Auto-generated by @wavehouse/sdk codegen +export interface Database { + clicks: ClicksRow; + events: EventsRow; +} + +export interface ClicksRow { + event_id: string; + page: string; + user_id: string; + duration_ms: number; + received_timestamp: string; +} +``` + +**ClickHouse → TypeScript type mapping:** + +| ClickHouse Type | TypeScript Type | +|----------------|-----------------| +| `String`, `FixedString`, `UUID`, `DateTime*`, `Date*`, `Enum*`, `IPv4/6` | `string` | +| `UInt*`, `Int*`, `Float*`, `Decimal*` | `number` | +| `Bool` | `boolean` | +| `Nullable(T)` | `T \| null` | +| `Array(T)` | `T[]` | +| `Map(K, V)` | `Record` | +| `LowCardinality(T)` | same as `T` | + +## E2E Testing + +The SDK doubles as the E2E integration test harness. Tests in `tests/e2e/sdk/` exercise the full pipeline (ingest → ClickHouse → query) through the SDK, validating both the backend and the client library in one pass. + +```bash +# Run all E2E tests: the orchestrator boots a ClickHouse testcontainer + +# the wavehouse-cov binary, then runs the SDK suite +make test-e2e +``` + +Test files live in `tests/e2e/sdk/`: `admin`, `auth`, `batching`, `cache`, `dlq`, `ingest`, `ndjson`, `query`, `streaming`, `stress` (each `*.test.ts`). + +See [Development Guide — E2E Tests via SDK](/development#e2e-tests-via-sdk) for architecture details and workflow tips. diff --git a/docs/src/content/docs/sdk/streaming.md b/docs/src/content/docs/sdk/streaming.md new file mode 100644 index 00000000..2a9868a5 --- /dev/null +++ b/docs/src/content/docs/sdk/streaming.md @@ -0,0 +1,160 @@ +--- +title: "SDK Streaming & Live Queries" +description: "Real-time SSE streams, client-side filtering, and backfill-then-live queries in @wavehouse/sdk." +--- + +Real-time consumption with `@wavehouse/sdk`: SSE event streams from tables, +builders, and pipes, plus live queries that backfill history before going +live. Builders and table refs come from [Queries](/sdk/queries). +Examples import from `@wavehouse/sdk`; using the CDN instead, import from +`https://esm.sh/@wavehouse/sdk` (see [Imports & Runtimes](/sdk#imports--runtimes)). + +## Streaming + +Streams use SSE (Server-Sent Events) for both unauthenticated connections and for authenticated ones. + +### `StreamController` + +Returned by `.stream()` on `TableRef`, `QueryBuilder`, `PipeRef`, and `DLQNamespace` (the DLQ variant is not yet functional server-side — [#197](https://github.com/Wave-RF/WaveHouse/issues/197)). It is **NOT thenable**. + +```ts +const stream = wh.from('clicks').stream({ since: '2026-01-01T00:00:00Z' }); +``` + +### `.subscribe(subscriber)` → `unsubscribe()` + +Callback-based consumption. Returns a cleanup function. + +```ts +const unsub = stream.subscribe({ + next: (event) => { + // event: { table: 'clicks', timestamp: '2026-...', data: { page: '/', ... } } + console.log('New event:', event.data); + }, + status: (state) => { + // state: 'connecting' | 'live' | 'reconnecting' | 'closed' + updateIndicator(state); + }, + error: (err) => { + console.error('Stream error:', err.message); + }, +}); + +// Cleanup — closes the connection if no other subscribers remain +unsub(); +``` + +### Async Iterator + +```ts +const stream = wh.from('clicks').stream(); + +for await (const event of stream) { + console.log(event.table, event.data); + if (shouldStop) break; // breaking auto-closes the stream +} +``` + +### `.close()` + +Explicitly close the stream and release all resources. + +```ts +stream.close(); +``` + +### `.status` + +Current connection status: `'connecting' | 'live' | 'reconnecting' | 'closed'`. + +### `StreamOptions` + +| Field | Type | Description | +| ----- | ---- | ----------- | +| `since` | `string` | RFC3339 timestamp for gap-fill replay | +| `signal` | `AbortSignal` | Cancel/close the stream. Wired via `attachSignal()` internally. | + +### `StreamEvent` + +```ts +interface StreamEvent { + table: string; // table name (e.g. 'clicks') + timestamp: string; // received_timestamp (RFC3339Nano) + data: T; // row data +} +``` + +### Transport Behavior + +| Transport | Reconnect | Protocol | +| --------- | --------- | -------- | +| SSE | Automatic (native `EventSource` with `Last-Event-ID`) | HTTP/2 recommended | + + + +:::note[SSE connection limit] +The SDK warns when more than 5 concurrent SSE connections are open (browser limit per domain). +::: + +### Client-Side Stream Filtering + +When a `QueryBuilder` with `.where()` filters or `.select()` columns calls `.stream()`, the returned stream applies those filters client-side: + +```ts +const stream = wh.from('clicks') + .select('page', 'button') + .where('page', '=', '/home') + .stream(); + +// Only events where page === '/home' are emitted, with only page + button columns +``` + +Supported operators: `=`, `!=`, `>`, `>=`, `<`, `<=`, `in`, `like`, `not_like` — the same `FilterOp` set `.where()` takes everywhere (the SDK maps them to wire tokens such as `eq`/`neq` internally). + +--- + +## Live Queries + +Live queries combine a historical backfill (`.fetch()`) with a real-time stream, providing a seamless initial load + live updates experience. + +```ts +const lq = wh.from('clicks') + .selectAll() + .where('page', '=', '/home') + .orderBy('received_timestamp', 'desc') + .limit(100) + .liveQuery({ + initial: (result) => { + // Called once with the historical data + setRows(result.data ?? []); + }, + next: (event) => { + // Called for each live event after backfill + addRow(event.data); + }, + error: (err) => console.error(err), + }); + +// Cleanup +lq.close(); +``` + +### `StreamSubscriber` + +```ts +interface StreamSubscriber { + initial?: (result: Result) => void; // Historical backfill result + next: (event: StreamEvent) => void; // Live events + status?: (state: string) => void; // Connection state changes + error?: (err: WaveHouseError) => void; // Errors +} +``` + +### How it works + +1. Opens the stream **immediately** and buffers incoming events. +2. Runs the `.fetch()` query for historical data, calls `subscriber.initial()` with the result. +3. Deduplicates buffered events by comparing timestamps against the latest historical timestamp. +4. Flushes remaining buffered events and switches to live mode. + +This "stream-first" approach ensures no events are lost between the fetch and stream start. diff --git a/docs/src/styles/global.css b/docs/src/styles/global.css index a985ae65..dbd000df 100644 --- a/docs/src/styles/global.css +++ b/docs/src/styles/global.css @@ -230,7 +230,9 @@ --sl-color-black: var(--wh-bg); --sl-color-bg: var(--wh-bg); - --sl-color-bg-nav: color-mix(in oklab, var(--wh-bg) 78%, transparent); + /* 88%, was 78% — scrolled body text stayed legible through the glass and + collided visually with the search trigger (fresh-eyes audit). */ + --sl-color-bg-nav: color-mix(in oklab, var(--wh-bg) 88%, transparent); --sl-color-bg-sidebar: var(--wh-bg); --sl-color-bg-inline-code: var(--wh-code-bg); --sl-color-hairline: var(--wh-border); @@ -247,14 +249,14 @@ --sl-text-h4: 1.125rem; --sl-text-h5: 1rem; - --sl-sidebar-width: 18.5rem; - /* Mobile/tablet default — fits a 768px iPad without clipping (the old 50rem - slightly overflowed there). Widened to 58rem only at the `lg` breakpoint - below, where the 3-column layout has room: code/tables/wide Mermaid then - use the space on large monitors while prose line length stays held by the - .sl-markdown-content text-block cap. The shell is also capped + centered - on ultrawide via .main-frame so content+TOC don't strand dead space. */ - --sl-content-width: 45rem; + /* Layout geometry stays at Starlight's stock values (18.75rem sidebar, + ONE 45rem content width at every breakpoint). A previous pass widened + the content track in steps (50rem @72rem, 58rem @100rem) and sub-capped + prose at 46rem — which left text, tables, and code all ending at + different right edges and shifted the column at each breakpoint. One + width keeps every block aligned and preserves the TOC's designed share + of the leftover space. Wide tables/diagrams scroll or scale inside + their own wrappers. */ --sl-nav-height: 4rem; --sl-shadow-sm: var(--wh-shadow-sm); @@ -272,24 +274,6 @@ --sl-color-gray-7: var(--wh-surface-elevated); } -/* Content track widens in two steps so it never starves the right-rail TOC - (content does NOT shrink below --sl-content-width in this layout, so an - over-wide track squeezes the TOC into ellipsis). 50rem through the laptop - range is exactly the pre-sweep width — no TOC regression there. The full - 58rem engages only ≥100rem, the same breakpoint where .main-frame caps + - centers the shell: at ≥1600px there's unambiguous room for sidebar + 58rem - content + a comfortable TOC, and the two changes reinforce each other. */ -@media (min-width: 72rem) { - :root { - --sl-content-width: 50rem; - } -} -@media (min-width: 100rem) { - :root { - --sl-content-width: 58rem; - } -} - /* ---------- Mermaid diagram polish ---------- These rules live at the document level (not in each SVG's inline