Skip to content

Commit 26b9866

Browse files
docs(deploy): document JetStream SyncAlways durability vs throughput (#338)
## What Adds a **Durability & Storage** operations page (`docs/src/content/docs/durability.md`) that makes the embedded-JetStream durability contract explicit, and wires it into the nav + cross-links. The core fact this documents: WaveHouse starts embedded NATS with `SyncAlways: true` (`internal/mq/embedded.go`), so **every event is `fsync`'d to disk before `POST /v1/ingest` returns `200`**. That makes the storage substrate's `fsync` tail the ingest latency floor — a deliberate, strong guarantee that was previously implicit and undocumented. ## Why Per [#84](#84), the durability/throughput tradeoff is a product decision worth documenting independently of the infra mitigation in support-infra#1. The issue author flagged this as the **docs-site-publish-gating "item"** ([comment](#84 (comment))): operators need to know what a `200` ack guarantees and how to tell whether their storage can sustain it before they go live. ## The new page covers - **The contract** — `200` = `fsync`'d to local disk, and how that differs from JetStream's default (ack from OS page cache, periodic background flush). - **Why the `fsync` tail is the ingest floor** — and why a slow substrate surfaces as `create stream: ... context deadline exceeded` at boot and `503` backpressure under load. - **Where `SyncAlways` is cheap vs. expensive** — managed cloud block storage / PLP NVMe / local ext4 vs. ZFS-without-SLOG / qcow2-on-ext4 / spinning disks. - **Measure your own storage** — an `fio` recipe + verdict bands (calibrated to the `SyncAlways` default), with the macOS `F_FULLFSYNC` honesty caveat. - **Symptom checklist** — what bad storage looks like in practice. ## Cross-links / wiring - Registered in `docs/src/config/sidebar.ts` under **Operations** (between Deployment and Behind a reverse proxy; reverse-proxy's `sidebar.order` bumped 11→12). - Cross-linked from **Configuration → Message Queue (NATS)**, **Deployment → Persistent Storage**, and **Ingest Pipeline → Backpressure and durability knobs** (which already documents the *worker-side* ack cost — this page owns the *producer-side* contract, no duplication). - `CHANGELOG.md` `[Unreleased] → Added`. ## Scope (and what's intentionally out of it) This PR is the **documentation deliverable** of #84. It does **not**: - Add the `mq.sync_interval` config knob — that's the code-side companion, tracked in **[#139](#139. The page forward-references it without claiming it exists yet. - Implement the proposed `wavehouse storage-check` CLI subcommand — a larger feature still tracked in #84. The page forward-references it as planned. I used **"Documents #84"** rather than "Closes #84" so the issue stays open to track those two remaining deliverables. Suggest splitting the `storage-check` CLI into its own issue and closing #84 once the docs land, or keeping it open until the CLI ships — reviewer's call. ## Testing - `make build-docs` ✅ — 21 pages built, `starlight-links-validator` clean (all new cross-link anchors resolve), `astro check` passes. - `make verify` ✅ (pre-commit) — markdownlint, misspell, astro content-schema check. - Docs-only change (`scripts/classify-paths.sh` → `code=false`), so no Go/SDK suites apply. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent 7c3b6d2 commit 26b9866

6 files changed

Lines changed: 110 additions & 1 deletion

File tree

‎CHANGELOG.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
1111

1212
### Added
1313

14+
- **"Durability & Storage" operations guide** (`docs/src/content/docs/durability.md` (new), `docs/src/config/sidebar.ts`, `docs/src/content/docs/reverse-proxy.mdx`, `docs/src/content/docs/configuration.mdx`, `docs/src/content/docs/deployment.md`): documents #84. A new Operations page making the embedded-JetStream durability contract explicit before the docs site publishes: a `200` from `POST /v1/ingest` means the event has been `fsync`'d to disk on the node (the server runs with `SyncAlways: true` in `internal/mq/embedded.go`), which makes the storage substrate's `fsync` tail the ingest latency floor. Covers the contract (and how it differs from JetStream's default page-cache-then-periodic-sync mode), why a slow `fsync` tail manifests as `create stream: ... context deadline exceeded` and `503` backpressure, a where-it's-cheap-vs-expensive substrate table (managed cloud block storage and PLP NVMe vs. ZFS-without-SLOG / qcow2-on-`ext4` / spinning disks), an `fio` recipe + verdict bands to measure your own storage (with the macOS `F_FULLFSYNC` honesty caveat), and the symptom checklist. Forward-references the configurable group-commit interval (`mq.sync_interval`, [#139](https://github.com/Wave-RF/WaveHouse/issues/139)) and the planned `wavehouse storage-check` preflight ([#84](https://github.com/Wave-RF/WaveHouse/issues/84)) without claiming either exists yet. Cross-linked from Configuration (Message Queue), Deployment (Persistent Storage), and the Ingest Pipeline's worker-side ack section; no code changes.
1415
- **"Behind a reverse proxy" deployment guide** (`docs/src/content/docs/reverse-proxy.mdx` (new), `docs/src/config/sidebar.ts`, `docs/src/content/docs/deployment.md`, `docs/src/content/docs/configuration.mdx`, `docs/src/content/docs/api.md`, `internal/api/stream.go`, `internal/api/stream_test.go`): closes #241. A new Operations page for the common "WaveHouse behind nginx / Caddy / Cloudflare Tunnel" setup, since several behaviors only matter behind a proxy and weren't documented together. Covers: TLS termination (WaveHouse serves plain HTTP and manages no certs); the request-body size limits and the division of responsibility (WaveHouse ships fixed in-code memory-safety backstops — 1 MiB control / 16 MiB ingest — while the proxy is the tunable *outer* limit, so a missing/loose proxy limit can't OOM the server); Server-Sent Events buffering + idle-timeout tuning (WaveHouse sends one `: connected` comment then **no periodic heartbeat** yet, [#226](https://github.com/Wave-RF/WaveHouse/issues/226), so quiet streams rely on the proxy not closing idle connections and on `EventSource` auto-reconnect); the `?token=` / `since` / `Last-Event-ID` forwarding streams need; `X-Forwarded-For` trust (don't expose `:8080` directly — it's honored, so a direct client could spoof it); and which health paths to expose (`/livez`/`/readyz` internal-optional, `/v1/health` must stay public). Ships full example nginx, Caddy, and Cloudflare-Tunnel configs, and is cross-linked from Deployment, Configuration, and the API reference. One small code change lands with it: the SSE endpoint (`GET /v1/stream`) now sets `X-Accel-Buffering: no` so nginx-class proxies stream events without buffering out of the box (nginx strips the header before the client sees it; Caddy/Cloudflare ignore it). The health-probe guidance is also upgraded from "optional" to a recommendation — keep the bare `/livez`/`/readyz`/`/healthz` paths internal (a public `/readyz` turns each hit into a ClickHouse `Ping`) and expose only `/v1/health` publicly.
1516
- **Coverage publishing — a self-hosted Go coverage README badge and GitHub Code Quality PR comments** (`.github/workflows/ci.yml`, `.github/actionlint.yaml` (new), `scripts/cov/main.go`, `scripts/ci/publish-badge.sh` (new), `.testcoverage.yml`, `go.mod`/`go.sum`, `README.md`, `AGENTS.md`, `docs/src/content/docs/development.md`): closes #133, now that the repo is public. Two published surfaces, both **non-gating** — `make cov`'s thresholds stay the only merge gate. (1) **README badge**: a new `cov badge` subcommand renders a [shields.io endpoint](https://shields.io/endpoint) JSON for the merged Go total using the *exact* number `threshold.total` gates (same `.testcoverage.yml` excludes), and a new non-gating `badge` job — the sole holder of `contents:write`, running only on trusted main — publishes it to an orphan `badges` branch via `scripts/ci/publish-badge.sh`, which the README reads over `raw.githubusercontent.com` (unrestricted for a public repo). (2) **PR comments**: the `coverage` job converts the merged Go profile to Cobertura (`go tool gocover-cobertura`, a new pinned Go `tool` dependency, with `-ignore-dirs` mirroring the YAML's global excludes) and uploads it to GitHub Code Quality via `actions/upload-code-coverage` (`code-quality: write`); the `github-code-quality[bot]` posts the aggregate + per-file diff-vs-`main` comment. The upload is `continue-on-error` so this public-preview GitHub feature can never red CI, and fork PRs skip it (no `code-quality` token, per GitHub's own guard). `actionlint` doesn't recognize the preview `code-quality` permission scope yet, so a new `.github/actionlint.yaml` suppresses only that one message. Requires the repo's *Settings → Code quality* enablement for the comments to render. Full design in `.github/workflows/README.md` §"Coverage publishing".
1617

‎docs/src/config/sidebar.ts‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -46,6 +46,7 @@ export const sidebar: StarlightUserConfig["sidebar"] = [
4646
items: [
4747
{ label: "Configuration", slug: "configuration" },
4848
{ label: "Deployment", slug: "deployment" },
49+
{ label: "Durability & Storage", slug: "durability" },
4950
{ label: "Behind a reverse proxy", slug: "reverse-proxy" },
5051
],
5152
},

‎docs/src/content/docs/configuration.mdx‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -64,6 +64,8 @@ The server speaks **plain HTTP** — there is no inbound-TLS setting. Terminate
6464
| `mq.gap_window_minutes` | `WH_MQ_GAP_WINDOW_MINUTES` | `15` | How many minutes of messages to retain in NATS for SSE gap-fill. The Active Sweeper will not purge messages newer than this window. |
6565
| `mq.max_bytes_gb` | `WH_MQ_MAX_BYTES_GB` | `50` | Maximum NATS JetStream stream size in GB. When full, new publishes are rejected with `DiscardNew` policy, triggering 503 backpressure on the ingest endpoint. |
6666

67+
**Durability.** The embedded server runs with JetStream `SyncAlways`, so every event is `fsync`'d to disk before `POST /v1/ingest` returns `200`. This makes your storage's `fsync` latency your ingest latency floor — see [Durability & Storage](/durability) to check whether your substrate can sustain it. There is no knob to relax this today ([#139](https://github.com/Wave-RF/WaveHouse/issues/139) tracks a configurable group-commit interval).
68+
6769
### Deduplication
6870

6971
| YAML Key | Env Var | Default | Description |

‎docs/src/content/docs/deployment.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -171,6 +171,8 @@ In a Docker / Podman / Kubernetes deployment, **`data_dir` must resolve to a hos
171171

172172
If `data_dir` resolves into the container's writable overlay layer instead, **JetStream state is wiped on every restart**: in-flight events are lost, gap-fill stops bridging restarts, and disk usage accumulates inside `/var/lib/docker` instead of the volume the operator chose.
173173

174+
Beyond persistence, the *speed* of that volume matters: JetStream `fsync`s every event to `<data_dir>/nats` before the ingest endpoint returns `200`, so the volume's `fsync` latency is your ingest latency floor. Managed cloud block storage handles this without thinking; commodity or virtualized substrates (ZFS without a SLOG, qcow2-on-`ext4`, spinning disks) can stall ingest with multi-second `fsync` tails. See [Durability & Storage](/durability) to measure yours before going live.
175+
174176
WaveHouse runs a simple existence check on startup and logs a `WARN` if `<data_dir>/nats` (or `<data_dir>/pebble` when dedupe is on) is missing or empty:
175177

176178
```text wrap=false
Lines changed: 103 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,103 @@
1+
---
2+
title: "Durability & Storage"
3+
description: "What a WaveHouse ingest ack guarantees, why embedded JetStream fsyncs every publish, and how to tell whether your storage substrate can sustain it."
4+
sidebar:
5+
order: 11
6+
---
7+
8+
WaveHouse buffers every ingested event in embedded NATS JetStream before the [ingest worker](/ingest-pipeline) drains it into ClickHouse. That buffer lives on disk at `<data_dir>/nats`, and **WaveHouse runs JetStream in its strictest durability mode**: every publish is `fsync`'d to non-volatile storage before the producer is acknowledged.
9+
10+
This is a deliberate, strong guarantee — but it makes your ingest latency a direct function of your storage's `fsync` latency. On managed cloud block storage that is effectively free; on some commodity or virtualized substrates the `fsync` tail balloons into seconds and ingest visibly suffers. This page explains the contract, where it is cheap versus expensive, and how to measure your storage before you trust it.
11+
12+
## The durability contract
13+
14+
The embedded server is started with `SyncAlways: true` (`internal/mq/embedded.go`). Concretely:
15+
16+
> When a client receives `200` from `POST /v1/ingest`, the event has already been `fsync`'d to disk on the WaveHouse node.
17+
18+
Ingestion is still [asynchronous](/architecture) end-to-end — the `200` means *durably buffered in JetStream*, not yet *written to ClickHouse* (the worker flushes to ClickHouse later, and the [worker's own ack](/ingest-pipeline#backpressure-and-durability-knobs) is what records "now in ClickHouse"). But the buffering step itself is hard-durable: an event that got a `200` survives an immediate, uncontrolled power loss on the node.
19+
20+
This is the strongest mode JetStream offers. It is stronger than the default, where a publish is acked once the write reaches the OS page cache and the data is flushed to disk later by a periodic background sync — fast, but a hard crash can lose the not-yet-flushed window.
21+
22+
| Mode | Ack means | Crash exposure | Throughput |
23+
| --- | --- | --- | --- |
24+
| **`SyncAlways` (WaveHouse today)** | data is `fsync`'d to disk | none for acked events | bounded by `fsync` latency |
25+
| Periodic group commit (default JetStream) | data is in the OS page cache | up to one sync interval of acked-but-unflushed events | bounded by memory/CPU |
26+
27+
WaveHouse does not currently expose a knob to relax this — `SyncAlways` is always on. Exposing a configurable group-commit interval (`mq.sync_interval`) is tracked in [#139](https://github.com/Wave-RF/WaveHouse/issues/139).
28+
29+
## Why the fsync tail is your ingest floor
30+
31+
Because the publish blocks on `fsync`, **your typical ingest latency is your storage's typical `fsync` latency, and your worst-case publish is your storage's worst-case `fsync`.** When that tail is healthy (sub-millisecond to single-digit milliseconds) the guarantee is essentially free. When it is not, the same code path that handles every production message stalls:
32+
33+
- Publishes block for the duration of the `fsync`, so a multi-second `fsync` tail is a multi-second ingest tail.
34+
- The embedded server's stream/consumer setup and every publish run under the JetStream client's request timeout; a slow-enough substrate makes them exceed it. The boot-time symptom is `create stream: ... context deadline exceeded`.
35+
- If the worker cannot drain to ClickHouse faster than producers publish, the stream fills toward `mq.max_bytes_gb` and the API returns `503` ([backpressure by construction](/ingest-pipeline#backpressure-and-durability-knobs)).
36+
37+
## Where `SyncAlways` is cheap vs. expensive
38+
39+
The strict guarantee translates well to managed cloud infrastructure — the presumed production target — and to enterprise-grade local disks. It is the commodity and virtualized substrates that bite.
40+
41+
**Healthy — `SyncAlways` is effectively free:**
42+
43+
| Substrate | Why |
44+
| --- | --- |
45+
| Cloud block storage (gp3/io2 EBS, GCP pd-ssd, Azure Premium SSD) | Battery-backed cache acks sync writes from non-volatile DRAM, not NAND. Sub-millisecond at typical load. |
46+
| Enterprise NVMe with power-loss protection (Optane, Samsung PM-series, Solidigm D7) | The PLP capacitor lets the controller ack a sync write from DRAM — the `fsync` ≈ `memcpy`. |
47+
| Local `ext4` on consumer NVMe | Single-device journal commit, 1–10 ms typical. Tail spikes under heavy concurrent dirty data, but bounded. |
48+
49+
**Problematic — measure before you trust it:**
50+
51+
| Substrate | Failure mode |
52+
| --- | --- |
53+
| ZFS without a SLOG, consumer NVMe | Every sync write hits the ZIL, gated by transaction-group commit cadence that serializes across all pool consumers. Single-digit-ms idle, **5–25 s under concurrent load**. |
54+
| Loopback / qcow2 on `ext4` inside a VM | Stacks a second journaling layer; often 10× slower than direct `ext4` and highly variable. |
55+
| Spinning disks | Mechanical seek on the NAND-equivalent program path: multi-millisecond baseline, multi-second tail. |
56+
57+
The tell for a commit-cadence problem (ZFS-without-SLOG, noisy-neighbor VM host) is that a single-threaded benchmark looks fine while a concurrent one is far worse — so always benchmark with multiple writers, and benchmark the guest **and** the host if virtualized.
58+
59+
## Check your storage before you trust it
60+
61+
Replicate JetStream's exact pattern — a 4 KiB write followed by a flush, in a tight loop — and report the percentiles. The numbers that matter are **p99** and **max**: those are your worst-case publish latency.
62+
63+
On Linux, [`fio`](https://fio.readthedocs.io/) (packaged on every distro) is the honest, standard tool. Point it at the volume that backs `<data_dir>/nats`, ideally before WaveHouse is running:
64+
65+
```bash
66+
# 8 concurrent writers — the variant that surfaces commit-cadence problems
67+
fio --name=jetstream-fsync --directory=/var/lib/wavehouse/nats \
68+
--rw=write --bs=4k --size=64M --fsync=1 --runtime=30 --time_based \
69+
--numjobs=8 --group_reporting
70+
```
71+
72+
Run it under representative load, not on an idle box — idle benchmarks understate real-world tails.
73+
74+
Read the measured p99 against these bands, which track WaveHouse's `SyncAlways` default:
75+
76+
| p99 `fsync` | Verdict for `SyncAlways: true` |
77+
| ---: | --- |
78+
| < 1 ms | **Ideal** |
79+
| 1–5 ms | **Good** |
80+
| 5–50 ms | **Workable** — watch bursty load |
81+
| 50 ms – 1 s | **Marginal** — relax durability once `mq.sync_interval` ([#139](https://github.com/Wave-RF/WaveHouse/issues/139)) lands, or move to faster storage |
82+
| > 1 s | **Broken** — `create stream` will time out under load; fix the storage substrate |
83+
84+
:::caution[macOS `fsync` lies by default]
85+
A plain `fsync()` on macOS returns once data is in the drive's volatile cache — it does **not** force a flush to NAND; only `fcntl(fd, F_FULLFSYNC)` does (NATS, Postgres, and SQLite all use it). On a Mac, any per-flush number under ~1 ms is almost certainly not a real flush — the gap between plain `fsync()` and `F_FULLFSYNC` can be ~180× on the same consumer NVMe. `fio` on macOS calls plain `fsync()`, so don't trust Mac `fio` numbers for tail-latency planning. This mostly matters when benchmarking a dev machine; production WaveHouse runs on Linux, where `fio` is honest.
86+
:::
87+
88+
A self-contained `wavehouse storage-check` preflight subcommand that bakes this measurement and verdict into the binary — including the per-platform honest flush — is tracked in [#84](https://github.com/Wave-RF/WaveHouse/issues/84).
89+
90+
## Symptoms of storage that can't keep up
91+
92+
If you see any of these, benchmark the `<data_dir>/nats` volume as above:
93+
94+
- `create stream: ... context deadline exceeded` at startup.
95+
- Ingest p99 latency in the seconds, or occasional `200`s that take multiple seconds to return.
96+
- Intermittent `503 Service Unavailable` from `/v1/ingest` when ClickHouse is healthy (the worker can't drain fast enough because acking is `fsync`-bound).
97+
- Flaky CI or load tests that pass on fast storage and fail on a shared/virtualized host.
98+
99+
## See also
100+
101+
- [Configuration → Message Queue (NATS)](/configuration#message-queue-nats) — the `mq.*` knobs (`gap_window_minutes`, `max_bytes_gb`).
102+
- [Deployment → Persistent Storage](/deployment#persistent-storage-required-for-containers) — `data_dir` must resolve to a host-backed volume.
103+
- [Ingest Pipeline → Backpressure and durability knobs](/ingest-pipeline#backpressure-and-durability-knobs) — the worker-side ack cost and the in-flight backpressure layers.

‎docs/src/content/docs/reverse-proxy.mdx‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22
title: "Behind a reverse proxy"
33
description: "Running WaveHouse behind nginx, Caddy, or a Cloudflare Tunnel: TLS, request-body limits, SSE buffering and timeouts, header/auth forwarding, and health probes."
44
sidebar:
5-
order: 11
5+
order: 12
66
---
77

88
import { Tabs, TabItem } from "@astrojs/starlight/components";

0 commit comments

Comments
 (0)