You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 86e041d
Browse filesBrowse the repository at this point in the historyBrowse files
Refines the SSE keepalive (#226) added in this PR and lays groundwork for
the delivery-path throughput epic (#294).
Config / behavior:
- Rename stream.heartbeat_* -> stream.keepalive_*. The knob is now the
effective per-connection period (keepalive_interval, default 30s); the
per-tick interval is derived as interval/keepalive_buckets, so one
rotation always spans the interval regardless of bucket count.
- 30s default clears the common 55-60s nginx/ingress-nginx/ALB/Heroku idle
windows with ~2x margin (was an effective 15s). 0 -> default, negative
rejected at startup.
- Minimal ":\n\n" keepalive payload (was ": heartbeat").
Refactor (groundwork for #294):
- Extract the wheel into a new internal/stream package, one abstraction per
file: Subscriber (subscriber.go), the Bucket fan-out primitive
(bucket.go), the Heartbeater wheel (heartbeat.go).
- Subscriber owns a generic ready-to-write outbound queue (Send/Frames), so
the handler's keepalive case is now a payload-agnostic byte-pump that
#294's projected events route through; the bespoke event-transform case
is annotated as the #294 collapse target.
Tests / docs:
- Add -race teardown coverage: concurrent Add/Remove/Push, push-after-
remove, push-to-stopped-reader, and a handler connect/disconnect race.
- New reverse-proxy "Idle timeouts by provider" reference (sourced
per-provider table + how-to-change, incl. absolute-cap exceptions a
keepalive can't fix) and doc-sync across AGENTS.md + api/config/
architecture/deployment + CHANGELOG.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XwhWrnP4q4634V3bjm1ZjJ
-**`auth/`** — JWT auth middleware: HMAC **or** JWKS verification with `alg` pinned to the active verifier, role extraction from a configurable claim path; always runs, never rejects (bad token → empty role + stashed reason)
@@ -41,6 +41,7 @@ Thirteen internal packages under `internal/` (plus `internal/testutil/` for shar
41
41
-**`pipes/`** — Named query pipes: `NamedQuery` type + NATS KV store (`WAVEHOUSE_PIPES`) + `.sql` file bootstrap
42
42
-**`policy/`** — Hasura-style access control: `Policy`/`TablePolicy`/`RolePermissions` types, `Evaluate()` engine with JWT claim templating, NATS KV store (`WAVEHOUSE_POLICY`)
Copy file name to clipboardExpand all lines: docs/src/content/docs/api.md
+2-2Lines changed: 2 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -516,7 +516,7 @@ Opens a persistent SSE connection for real-time event streaming. Supports histor
516
516
| ------ | ----------- |
517
517
|`Last-Event-ID`| RFC 3339 timestamp of the last received event. If present, overrides the `since` query parameter for automatic reconnection (standard `EventSource` behavior). |
518
518
519
-
**Response:** SSE stream (`text/event-stream`). Each event includes an `id:` field set to the event's `received_timestamp`. The stream opens with a `: connected` comment and emits a `: heartbeat`comment periodically (about every 15 seconds by default), which keeps a quiet connection from being closed by a proxy; both are standard SSE comments that `EventSource` ignores (raw consumers should skip `:`-prefixed lines).
519
+
**Response:** SSE stream (`text/event-stream`). Each event includes an `id:` field set to the event's `received_timestamp`. The stream opens with a `: connected` comment and emits a minimal `:` keepalive comment periodically (every 30 seconds by default), which keeps a quiet connection from being closed by a proxy; both are standard SSE comments that `EventSource` ignores (raw consumers should skip `:`-prefixed lines).
520
520
521
521
```text
522
522
id: 2026-03-24T12:00:00.123Z
@@ -533,7 +533,7 @@ Each SSE connection is bound to a single `?table=`; to consume multiple tables,
533
533
**CORS:**`/v1/stream` honors the `server.cors_allowed_origins` allowlist like every endpoint, so a browser `EventSource` from an allowed origin connects normally. `Last-Event-ID` is allow-listed in the CORS preflight so fetch-based clients can resume cross-origin.
534
534
535
535
:::caution[Behind a proxy: disable response buffering]
536
-
SSE needs one bit of proxy configuration: disable response buffering, or the proxy holds events until a buffer fills and clients receive nothing in real time. Idle timeouts are handled for you — the `: heartbeat` comment above keeps a quiet stream alive under typical proxy/tunnel idle windows ([#226](https://github.com/Wave-RF/WaveHouse/issues/226)), so raising the idle/read timeout is now optional. Browser `EventSource` still auto-reconnects (resuming via `Last-Event-ID`) if a connection drops. See [Behind a reverse proxy → Server-Sent Events](/reverse-proxy#server-sent-events-sse) for nginx/Caddy/Cloudflare specifics.
536
+
SSE needs one bit of proxy configuration: disable response buffering, or the proxy holds events until a buffer fills and clients receive nothing in real time. Idle timeouts are handled for you — the `:` keepalive comment above keeps a quiet stream alive under typical proxy/tunnel idle windows ([#226](https://github.com/Wave-RF/WaveHouse/issues/226)), so raising the idle/read timeout is now optional. Browser `EventSource` still auto-reconnects (resuming via `Last-Event-ID`) if a connection drops. See [Behind a reverse proxy → Server-Sent Events](/reverse-proxy#server-sent-events-sse) for nginx/Caddy/Cloudflare specifics.
Copy file name to clipboardExpand all lines: docs/src/content/docs/architecture.md
+9-2Lines changed: 9 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -75,14 +75,21 @@ The API layer uses [Chi](https://github.com/go-chi/chi) for routing with Request
75
75
-**structured_query.go** — Handler for `POST /v1/query?table={table}`: validates query AST, enforces permissions, builds and executes SQL.
76
76
-**ingest.go** — Accepts flat JSON body for `POST /v1/ingest?table={table}`, validates against discovered schema, optional dedup, publishes to NATS subject `ingest.{table}`.
77
77
-**query.go** — Proxies raw SQL for `POST /v1/admin/query` straight to ClickHouse's HTTP interface. **Not cached** — sets `Cache-Control: no-store` so every request hits ClickHouse; DateTime is rendered ISO-8601 via `date_time_output_format=iso` (the Go-side type conversion lives in the structured-query / pipes path, not here).
78
-
-**stream.go** — Real-time streaming via SSE. Callers select a table with the `?table=` query parameter. Supports gap-fill from NATS JetStream using `DeliverByStartTime`. Each connection registers with the shared `Heartbeater` (see `heartbeat.go`) so idle streams keep emitting `: heartbeat` comments and survive reverse-proxy idle timeouts.
79
-
-**heartbeat.go** — Shared SSE keepalive: one process-wide `Heartbeater` ticker fans `: heartbeat` comments out across a ring of `Bucket`s, waking ~1/N of live streams per tick so the writes don't synchronize. Per-connection period is `heartbeat_interval × heartbeat_buckets`; the owning handler goroutine does the actual write, so the shared ticker never touches a `ResponseWriter` directly.
78
+
-**stream.go** — Real-time streaming via SSE. Callers select a table with the `?table=` query parameter. Supports gap-fill from NATS JetStream using `DeliverByStartTime`. Each connection registers with the shared keepalive wheel (the `stream/` package) so idle streams keep emitting `:` keepalive comments and survive reverse-proxy idle timeouts.
80
79
-**transform.go** — Shared `transformForClient` function: passes through `table_name`, `received_timestamp`, and `data` from the wire format.
81
80
-**schema.go** — Schema discovery API: list all schemas, get one table, trigger refresh.
82
81
-**dlq.go** — DLQ stats endpoint and `EnsureDLQStream` helper for creating the `WAVEHOUSE_DLQ` NATS stream.
83
82
-**hub.go** — In-process pub/sub for broadcasting MQ messages to connected streaming clients.
84
83
-**health.go** — Liveness (`/livez`), readiness (`/readyz`), and a content-free `Online` ping (`/v1/health`, the SDK's public liveness check); `/healthz` is a permanent alias of `/livez`, and `/health`/`/ready` are deprecated aliases. All three consult an optional `BootState` so they can return 503 while boot-time schema discovery is still failing in the retry loop (see `cmd/wavehouse/main.go`); once `BootState.Set(nil)` fires, `/livez` returns 200 and stays there. `/readyz` additionally pings ClickHouse each call; `/v1/health` deliberately does not.
85
84
85
+
### `stream/` — SSE keepalive & fan-out
86
+
87
+
The SSE fan-out primitives shared by the streaming API, factored out of `api/` so the delivery-path throughput work ([#294](https://github.com/Wave-RF/WaveHouse/issues/294)) can reuse them. One abstraction per file.
88
+
89
+
-**subscriber.go** — `Subscriber`, the per-connection handle. It owns a single ready-to-write outbound queue: producers fan bytes in with `Send` (non-blocking, drop-on-full), and the API handler writes whatever arrives on `Frames()` to the client verbatim. Today the only producer is the keepalive wheel and the only frames are `:` comments; #294 routes projected event frames through the same `Send`/`Frames` queue (and grows the handle with drop metrics), at which point the handler's keepalive and event cases collapse into one byte-pump.
90
+
-**bucket.go** — `Bucket`, the reusable fan-out primitive: a concurrency-safe set of subscribers that one `Push` delivers a shared byte slice to (snapshot-under-read-lock, then `Send` to each). The keepalive wheel holds a ring of them for load-spreading; #294 holds one per (role, table) column-set so a projected frame is built once and Push'd to every member instead of re-projected per subscriber.
91
+
-**heartbeat.go** — The keepalive wheel (`Heartbeater`). A single process-wide ticker fans a minimal `:` comment across the ring of `Bucket`s, waking ~1/N of live streams per tick so the writes don't synchronize. The effective per-connection keepalive period is `stream.keepalive_interval` (the wheel ticks every `keepalive_interval ÷ keepalive_buckets`, so one rotation spans the interval); the owning handler goroutine does the actual write, so the shared ticker never touches a `ResponseWriter` directly.
92
+
86
93
### `auth/` — Authentication
87
94
88
95
-**auth.go** — `Middleware(cfg)`: the JWT auth middleware. Verifies tokens with HMAC **or** JWKS (never both), with the accepted `alg` pinned to the active verifier and checked before any key is consulted (rejects `alg: none` and cross-family confusion). Extracts the caller's role from a configurable dot-path claim (`auth.role_claim`, default `role`). It always runs and never rejects — a missing/invalid/expired token yields an empty role (resolved to `default_role` downstream), with the token error stashed in context so a denying gate can fail loud (`401`, not a bare `403`).
Copy file name to clipboardExpand all lines: docs/src/content/docs/configuration.mdx
+7-7Lines changed: 7 additions & 7 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -43,10 +43,10 @@ The server speaks **plain HTTP** — there is no inbound-TLS setting. Terminate
43
43
44
44
| YAML Key | Env Var | Default | Description |
45
45
| --- | --- | ------- | ----------- |
46
-
|`stream.heartbeat_interval`|`WH_STREAM_HEARTBEAT_INTERVAL`|`5s`|How often the shared heartbeat ticker fires, pushing a `: heartbeat` keepalive comment to one bucket of live `GET /v1/stream` connections. |
47
-
|`stream.heartbeat_buckets`|`WH_STREAM_HEARTBEAT_BUCKETS`|`3`|How many buckets the ticker spreads connections across, so each tick wakes only ~1/N of live streams instead of all at once. |
46
+
|`stream.keepalive_interval`|`WH_STREAM_KEEPALIVE_INTERVAL`|`30s`|Effective per-connection keepalive period — the longest a quiet `GET /v1/stream` connection goes without a write before the server sends a `:` keepalive comment. Keep it under your proxy/tunnel idle timeout. |
47
+
|`stream.keepalive_buckets`|`WH_STREAM_KEEPALIVE_BUCKETS`|`3`|Advanced load-spreading knob: connections are spread across N buckets so each internal tick (`keepalive_interval ÷ keepalive_buckets`) nudges only ~1/N of live streams instead of all at once. Most deployments leave this at `3`. |
48
48
49
-
The **effective per-connection heartbeat period is `heartbeat_interval × heartbeat_buckets`** (default 5s × 3 = 15s) — a connection is pinged once per full rotation of the ring. Keep that product comfortably under your reverse proxy or tunnel's idle timeout (nginx's default `proxy_read_timeout` is 60s; Cloudflare's edge is ~100s) so quiet streams aren't dropped — see [Behind a reverse proxy → Server-Sent Events](/reverse-proxy#server-sent-events-sse). Each tick wakes only one bucket, so more buckets spread the write load across more ticks **but also lengthen the period** (it's the product above): raising `heartbeat_buckets` alone pushes the period out, so lower `heartbeat_interval` proportionally if you want more buckets at the same period. A value of `0` falls back to the default; a negative value is rejected at startup.
49
+
`keepalive_interval` is the number you tune: the **maximum time a connection can sit idle before it's nudged**. The default **30s** clears the common 55–60s proxy/load-balancer idle windows (nginx and ingress-nginx `proxy_read_timeout`, AWS ALB, Heroku) with roughly 2× safety margin, and comfortably clears Cloudflare's ~100s edge. Lower it if you front WaveHouse with something tighter — Azure Application Gateway (20s) or CloudFront (30s) — see [Behind a reverse proxy → Idle timeouts](/reverse-proxy#idle-timeouts-by-provider) for a per-provider table. `keepalive_buckets` only changes how the writes are *spread in time*, not the period: the wheel fires one bucket every `keepalive_interval ÷ keepalive_buckets`, so a full rotation always spans exactly `keepalive_interval` regardless of bucket count. A value of `0` for either falls back to the default; a negative value is rejected at startup.
Copy file name to clipboardExpand all lines: docs/src/content/docs/deployment.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -329,7 +329,7 @@ Until `startupProbe` succeeds, kubelet doesn't run `livenessProbe` or `readiness
329
329
330
330
## Behind a reverse proxy
331
331
332
-
WaveHouse serves plain HTTP on `:8080` and does **not** terminate TLS, manage certificates, or rate-limit — put a reverse proxy, CDN, or tunnel (nginx, Caddy, Cloudflare Tunnel) in front for any internet-facing deployment. A few behaviors only matter behind a proxy: TLS termination, the request-body size limits, Server-Sent Events buffering (WaveHouse now sends keepalive heartbeats so quiet streams survive proxy idle timeouts, [#226](https://github.com/Wave-RF/WaveHouse/issues/226)), header/auth forwarding, and which health paths to expose. See **[Behind a reverse proxy](/reverse-proxy)** for the full guide and example nginx/Caddy/Cloudflare configs.
332
+
WaveHouse serves plain HTTP on `:8080` and does **not** terminate TLS, manage certificates, or rate-limit — put a reverse proxy, CDN, or tunnel (nginx, Caddy, Cloudflare Tunnel) in front for any internet-facing deployment. A few behaviors only matter behind a proxy: TLS termination, the request-body size limits, Server-Sent Events buffering (WaveHouse now sends keepalive comments so quiet streams survive proxy idle timeouts, [#226](https://github.com/Wave-RF/WaveHouse/issues/226)), header/auth forwarding, and which health paths to expose. See **[Behind a reverse proxy](/reverse-proxy)** for the full guide and example nginx/Caddy/Cloudflare configs.
0 commit comments