Skip to content

Commit 86e041d

Browse files
EricAndrechekclaude
andcommitted
refactor(stream): keepalive package + generic Subscriber, 30s default
Refines the SSE keepalive (#226) added in this PR and lays groundwork for the delivery-path throughput epic (#294). Config / behavior: - Rename stream.heartbeat_* -> stream.keepalive_*. The knob is now the effective per-connection period (keepalive_interval, default 30s); the per-tick interval is derived as interval/keepalive_buckets, so one rotation always spans the interval regardless of bucket count. - 30s default clears the common 55-60s nginx/ingress-nginx/ALB/Heroku idle windows with ~2x margin (was an effective 15s). 0 -> default, negative rejected at startup. - Minimal ":\n\n" keepalive payload (was ": heartbeat"). Refactor (groundwork for #294): - Extract the wheel into a new internal/stream package, one abstraction per file: Subscriber (subscriber.go), the Bucket fan-out primitive (bucket.go), the Heartbeater wheel (heartbeat.go). - Subscriber owns a generic ready-to-write outbound queue (Send/Frames), so the handler's keepalive case is now a payload-agnostic byte-pump that #294's projected events route through; the bespoke event-transform case is annotated as the #294 collapse target. Tests / docs: - Add -race teardown coverage: concurrent Add/Remove/Push, push-after- remove, push-to-stopped-reader, and a handler connect/disconnect race. - New reverse-proxy "Idle timeouts by provider" reference (sourced per-provider table + how-to-change, incl. absolute-cap exceptions a keepalive can't fix) and doc-sync across AGENTS.md + api/config/ architecture/deployment + CHANGELOG. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XwhWrnP4q4634V3bjm1ZjJ
1 parent f0ce75d commit 86e041d

22 files changed

Lines changed: 734 additions & 375 deletions

‎AGENTS.md‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -26,7 +26,7 @@ One binary:
2626

2727
- **`cmd/wavehouse/`** — Standalone mode (all-in-one with embedded NATS, optional Pebble dedup)
2828

29-
Thirteen internal packages under `internal/` (plus `internal/testutil/` for shared test helpers):
29+
Fourteen internal packages under `internal/` (plus `internal/testutil/` for shared test helpers):
3030

3131
- **`api/`** — Chi HTTP router, JWT/JWKS middleware (from `auth/`), ingest/query/structured-query/SSE/schema/DLQ/policy/pipes handlers, Hub
3232
- **`auth/`** — JWT auth middleware: HMAC **or** JWKS verification with `alg` pinned to the active verifier, role extraction from a configurable claim path; always runs, never rejects (bad token → empty role + stashed reason)
@@ -41,6 +41,7 @@ Thirteen internal packages under `internal/` (plus `internal/testutil/` for shar
4141
- **`pipes/`** — Named query pipes: `NamedQuery` type + NATS KV store (`WAVEHOUSE_PIPES`) + `.sql` file bootstrap
4242
- **`policy/`** — Hasura-style access control: `Policy`/`TablePolicy`/`RolePermissions` types, `Evaluate()` engine with JWT claim templating, NATS KV store (`WAVEHOUSE_POLICY`)
4343
- **`query/`** — Structured query AST types + SQL builder with schema validation, permission injection, timestamp bucketing
44+
- **`stream/`** — SSE fan-out primitives: `Subscriber` (per-connection outbound frame queue, `Send`/`Frames`), the `Bucket` fan-out set (`subscriberSet`, reused by #294's delivery path), and the `Heartbeater` keepalive wheel
4445

4546
## Key Design Decisions
4647

@@ -399,6 +400,7 @@ internal/observability/ → OpenTelemetry pipeline (traces/metrics/logs provider
399400
internal/pipes/ → Named query pipes (NATS KV store + SQL file bootstrap)
400401
internal/policy/ → Access control policies (types, evaluation, NATS KV store)
401402
internal/query/ → Structured query AST + SQL builder
403+
internal/stream/ → SSE fan-out primitives (Subscriber outbound queue, Bucket fan-out, keepalive Heartbeater wheel)
402404
internal/testutil/ → Shared test helpers (NopLogger, etc.)
403405
tests/ → Integration & E2E tests
404406
tests/integration/ → Go integration tests (//go:build integration; ClickHouse testcontainer)

‎CHANGELOG.md‎

Lines changed: 2 additions & 2 deletions
Large diffs are not rendered by default.

‎cmd/wavehouse/main.go‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,7 @@ import (
2727
"github.com/Wave-RF/WaveHouse/internal/observability"
2828
"github.com/Wave-RF/WaveHouse/internal/pipes"
2929
"github.com/Wave-RF/WaveHouse/internal/policy"
30+
"github.com/Wave-RF/WaveHouse/internal/stream"
3031
)
3132

3233
// Pre-populated build info variables, set via ldflags in the Makefile.
@@ -359,7 +360,7 @@ func run() int {
359360
streamHandler := api.NewStreamHandler(hub, js)
360361
streamHandler.PolicyStore = policyStore
361362

362-
heartbeater := api.NewHeartbeater(cfg.Stream.HeartbeatBuckets, cfg.Stream.HeartbeatInterval)
363+
heartbeater := stream.NewHeartbeater(cfg.Stream.KeepaliveInterval, cfg.Stream.KeepaliveBuckets)
363364
go heartbeater.Run(ctx)
364365
streamHandler.Heartbeater = heartbeater
365366

‎config.yaml‎

Lines changed: 7 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -54,13 +54,14 @@ clickhouse:
5454
query:
5555
default_max_rows: 10000 # fallback result LIMIT (<=0 falls back to 10000)
5656

57-
# Server-Sent Events keepalive for GET /v1/stream. A shared ticker pushes a
58-
# ": heartbeat" comment so idle streams survive reverse-proxy/tunnel idle
59-
# timeouts. Effective per-connection period = heartbeat_interval × heartbeat_buckets
60-
# (default 15s); keep it under your proxy's idle timeout. See docs/reverse-proxy.
57+
# Server-Sent Events keepalive for GET /v1/stream. A shared ticker pushes a ":"
58+
# comment to idle streams so they survive reverse-proxy/tunnel idle timeouts.
59+
# keepalive_interval is the effective per-connection period; keep it under your
60+
# proxy's idle window (30s clears the common 55-60s nginx/ALB/Heroku timeouts).
61+
# See docs/reverse-proxy for per-provider timeout values.
6162
stream:
62-
heartbeat_interval: 5s # how often the shared ticker fires (0 falls back to 5s)
63-
heartbeat_buckets: 3 # connections spread across N buckets (0 falls back to 3)
63+
keepalive_interval: 30s # max idle time before a keepalive (0 falls back to 30s)
64+
keepalive_buckets: 3 # spread writes across N buckets (advanced; 0 falls back to 3)
6465

6566
mq:
6667
gap_window_minutes: 15

‎docs/src/content/docs/api.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -516,7 +516,7 @@ Opens a persistent SSE connection for real-time event streaming. Supports histor
516516
| ------ | ----------- |
517517
| `Last-Event-ID` | RFC 3339 timestamp of the last received event. If present, overrides the `since` query parameter for automatic reconnection (standard `EventSource` behavior). |
518518

519-
**Response:** SSE stream (`text/event-stream`). Each event includes an `id:` field set to the event's `received_timestamp`. The stream opens with a `: connected` comment and emits a `: heartbeat` comment periodically (about every 15 seconds by default), which keeps a quiet connection from being closed by a proxy; both are standard SSE comments that `EventSource` ignores (raw consumers should skip `:`-prefixed lines).
519+
**Response:** SSE stream (`text/event-stream`). Each event includes an `id:` field set to the event's `received_timestamp`. The stream opens with a `: connected` comment and emits a minimal `:` keepalive comment periodically (every 30 seconds by default), which keeps a quiet connection from being closed by a proxy; both are standard SSE comments that `EventSource` ignores (raw consumers should skip `:`-prefixed lines).
520520

521521
```text
522522
id: 2026-03-24T12:00:00.123Z
@@ -533,7 +533,7 @@ Each SSE connection is bound to a single `?table=`; to consume multiple tables,
533533
**CORS:** `/v1/stream` honors the `server.cors_allowed_origins` allowlist like every endpoint, so a browser `EventSource` from an allowed origin connects normally. `Last-Event-ID` is allow-listed in the CORS preflight so fetch-based clients can resume cross-origin.
534534

535535
:::caution[Behind a proxy: disable response buffering]
536-
SSE needs one bit of proxy configuration: disable response buffering, or the proxy holds events until a buffer fills and clients receive nothing in real time. Idle timeouts are handled for you — the `: heartbeat` comment above keeps a quiet stream alive under typical proxy/tunnel idle windows ([#226](https://github.com/Wave-RF/WaveHouse/issues/226)), so raising the idle/read timeout is now optional. Browser `EventSource` still auto-reconnects (resuming via `Last-Event-ID`) if a connection drops. See [Behind a reverse proxy → Server-Sent Events](/reverse-proxy#server-sent-events-sse) for nginx/Caddy/Cloudflare specifics.
536+
SSE needs one bit of proxy configuration: disable response buffering, or the proxy holds events until a buffer fills and clients receive nothing in real time. Idle timeouts are handled for you — the `:` keepalive comment above keeps a quiet stream alive under typical proxy/tunnel idle windows ([#226](https://github.com/Wave-RF/WaveHouse/issues/226)), so raising the idle/read timeout is now optional. Browser `EventSource` still auto-reconnects (resuming via `Last-Event-ID`) if a connection drops. See [Behind a reverse proxy → Server-Sent Events](/reverse-proxy#server-sent-events-sse) for nginx/Caddy/Cloudflare specifics.
537537
:::
538538

539539
**curl example:**

‎docs/src/content/docs/architecture.md‎

Lines changed: 9 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -75,14 +75,21 @@ The API layer uses [Chi](https://github.com/go-chi/chi) for routing with Request
7575
- **structured_query.go** — Handler for `POST /v1/query?table={table}`: validates query AST, enforces permissions, builds and executes SQL.
7676
- **ingest.go** — Accepts flat JSON body for `POST /v1/ingest?table={table}`, validates against discovered schema, optional dedup, publishes to NATS subject `ingest.{table}`.
7777
- **query.go** — Proxies raw SQL for `POST /v1/admin/query` straight to ClickHouse's HTTP interface. **Not cached** — sets `Cache-Control: no-store` so every request hits ClickHouse; DateTime is rendered ISO-8601 via `date_time_output_format=iso` (the Go-side type conversion lives in the structured-query / pipes path, not here).
78-
- **stream.go** — Real-time streaming via SSE. Callers select a table with the `?table=` query parameter. Supports gap-fill from NATS JetStream using `DeliverByStartTime`. Each connection registers with the shared `Heartbeater` (see `heartbeat.go`) so idle streams keep emitting `: heartbeat` comments and survive reverse-proxy idle timeouts.
79-
- **heartbeat.go** — Shared SSE keepalive: one process-wide `Heartbeater` ticker fans `: heartbeat` comments out across a ring of `Bucket`s, waking ~1/N of live streams per tick so the writes don't synchronize. Per-connection period is `heartbeat_interval × heartbeat_buckets`; the owning handler goroutine does the actual write, so the shared ticker never touches a `ResponseWriter` directly.
78+
- **stream.go** — Real-time streaming via SSE. Callers select a table with the `?table=` query parameter. Supports gap-fill from NATS JetStream using `DeliverByStartTime`. Each connection registers with the shared keepalive wheel (the `stream/` package) so idle streams keep emitting `:` keepalive comments and survive reverse-proxy idle timeouts.
8079
- **transform.go** — Shared `transformForClient` function: passes through `table_name`, `received_timestamp`, and `data` from the wire format.
8180
- **schema.go** — Schema discovery API: list all schemas, get one table, trigger refresh.
8281
- **dlq.go** — DLQ stats endpoint and `EnsureDLQStream` helper for creating the `WAVEHOUSE_DLQ` NATS stream.
8382
- **hub.go** — In-process pub/sub for broadcasting MQ messages to connected streaming clients.
8483
- **health.go** — Liveness (`/livez`), readiness (`/readyz`), and a content-free `Online` ping (`/v1/health`, the SDK's public liveness check); `/healthz` is a permanent alias of `/livez`, and `/health`/`/ready` are deprecated aliases. All three consult an optional `BootState` so they can return 503 while boot-time schema discovery is still failing in the retry loop (see `cmd/wavehouse/main.go`); once `BootState.Set(nil)` fires, `/livez` returns 200 and stays there. `/readyz` additionally pings ClickHouse each call; `/v1/health` deliberately does not.
8584

85+
### `stream/` — SSE keepalive & fan-out
86+
87+
The SSE fan-out primitives shared by the streaming API, factored out of `api/` so the delivery-path throughput work ([#294](https://github.com/Wave-RF/WaveHouse/issues/294)) can reuse them. One abstraction per file.
88+
89+
- **subscriber.go** — `Subscriber`, the per-connection handle. It owns a single ready-to-write outbound queue: producers fan bytes in with `Send` (non-blocking, drop-on-full), and the API handler writes whatever arrives on `Frames()` to the client verbatim. Today the only producer is the keepalive wheel and the only frames are `:` comments; #294 routes projected event frames through the same `Send`/`Frames` queue (and grows the handle with drop metrics), at which point the handler's keepalive and event cases collapse into one byte-pump.
90+
- **bucket.go** — `Bucket`, the reusable fan-out primitive: a concurrency-safe set of subscribers that one `Push` delivers a shared byte slice to (snapshot-under-read-lock, then `Send` to each). The keepalive wheel holds a ring of them for load-spreading; #294 holds one per (role, table) column-set so a projected frame is built once and Push'd to every member instead of re-projected per subscriber.
91+
- **heartbeat.go** — The keepalive wheel (`Heartbeater`). A single process-wide ticker fans a minimal `:` comment across the ring of `Bucket`s, waking ~1/N of live streams per tick so the writes don't synchronize. The effective per-connection keepalive period is `stream.keepalive_interval` (the wheel ticks every `keepalive_interval ÷ keepalive_buckets`, so one rotation spans the interval); the owning handler goroutine does the actual write, so the shared ticker never touches a `ResponseWriter` directly.
92+
8693
### `auth/` — Authentication
8794

8895
- **auth.go** — `Middleware(cfg)`: the JWT auth middleware. Verifies tokens with HMAC **or** JWKS (never both), with the accepted `alg` pinned to the active verifier and checked before any key is consulted (rejects `alg: none` and cross-family confusion). Extracts the caller's role from a configurable dot-path claim (`auth.role_claim`, default `role`). It always runs and never rejects — a missing/invalid/expired token yields an empty role (resolved to `default_role` downstream), with the token error stashed in context so a denying gate can fail loud (`401`, not a bare `403`).

‎docs/src/content/docs/configuration.mdx‎

Lines changed: 7 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -43,10 +43,10 @@ The server speaks **plain HTTP** — there is no inbound-TLS setting. Terminate
4343

4444
| YAML Key | Env Var | Default | Description |
4545
| --- | --- | ------- | ----------- |
46-
| `stream.heartbeat_interval` | `WH_STREAM_HEARTBEAT_INTERVAL` | `5s` | How often the shared heartbeat ticker fires, pushing a `: heartbeat` keepalive comment to one bucket of live `GET /v1/stream` connections. |
47-
| `stream.heartbeat_buckets` | `WH_STREAM_HEARTBEAT_BUCKETS` | `3` | How many buckets the ticker spreads connections across, so each tick wakes only ~1/N of live streams instead of all at once. |
46+
| `stream.keepalive_interval` | `WH_STREAM_KEEPALIVE_INTERVAL` | `30s` | Effective per-connection keepalive period — the longest a quiet `GET /v1/stream` connection goes without a write before the server sends a `:` keepalive comment. Keep it under your proxy/tunnel idle timeout. |
47+
| `stream.keepalive_buckets` | `WH_STREAM_KEEPALIVE_BUCKETS` | `3` | Advanced load-spreading knob: connections are spread across N buckets so each internal tick (`keepalive_interval ÷ keepalive_buckets`) nudges only ~1/N of live streams instead of all at once. Most deployments leave this at `3`. |
4848

49-
The **effective per-connection heartbeat period is `heartbeat_interval × heartbeat_buckets`** (default 5s × 3 = 15s) — a connection is pinged once per full rotation of the ring. Keep that product comfortably under your reverse proxy or tunnel's idle timeout (nginx's default `proxy_read_timeout` is 60s; Cloudflare's edge is ~100s) so quiet streams aren't dropped — see [Behind a reverse proxy → Server-Sent Events](/reverse-proxy#server-sent-events-sse). Each tick wakes only one bucket, so more buckets spread the write load across more ticks **but also lengthen the period** (it's the product above): raising `heartbeat_buckets` alone pushes the period out, so lower `heartbeat_interval` proportionally if you want more buckets at the same period. A value of `0` falls back to the default; a negative value is rejected at startup.
49+
`keepalive_interval` is the number you tune: the **maximum time a connection can sit idle before it's nudged**. The default **30s** clears the common 55–60s proxy/load-balancer idle windows (nginx and ingress-nginx `proxy_read_timeout`, AWS ALB, Heroku) with roughly 2× safety margin, and comfortably clears Cloudflare's ~100s edge. Lower it if you front WaveHouse with something tighter — Azure Application Gateway (20s) or CloudFront (30s) — see [Behind a reverse proxy → Idle timeouts](/reverse-proxy#idle-timeouts-by-provider) for a per-provider table. `keepalive_buckets` only changes how the writes are *spread in time*, not the period: the wheel fires one bucket every `keepalive_interval ÷ keepalive_buckets`, so a full rotation always spans exactly `keepalive_interval` regardless of bucket count. A value of `0` for either falls back to the default; a negative value is rejected at startup.
5050

5151
### ClickHouse
5252

@@ -221,8 +221,8 @@ server:
221221
- "*"
222222

223223
stream:
224-
heartbeat_interval: 5s # effective keepalive period = interval × buckets (default 15s)
225-
heartbeat_buckets: 3 # keep interval × buckets under your proxy's idle window
224+
keepalive_interval: 30s # max idle time before a keepalive; keep under your proxy's idle window
225+
keepalive_buckets: 3 # advanced: spread keepalive writes across N buckets
226226

227227
clickhouse:
228228
addr: localhost:9000
@@ -294,8 +294,8 @@ WH_SERVER_PORT=8080
294294
WH_SERVER_SHUTDOWN_TIMEOUT=10
295295
WH_SERVER_CORS_ALLOWED_ORIGINS=*
296296

297-
WH_STREAM_HEARTBEAT_INTERVAL=5s
298-
WH_STREAM_HEARTBEAT_BUCKETS=3
297+
WH_STREAM_KEEPALIVE_INTERVAL=30s
298+
WH_STREAM_KEEPALIVE_BUCKETS=3
299299

300300
WH_CH_ADDR=localhost:9000
301301
WH_CH_HTTP_PORT=8123

‎docs/src/content/docs/deployment.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -329,7 +329,7 @@ Until `startupProbe` succeeds, kubelet doesn't run `livenessProbe` or `readiness
329329

330330
## Behind a reverse proxy
331331

332-
WaveHouse serves plain HTTP on `:8080` and does **not** terminate TLS, manage certificates, or rate-limit — put a reverse proxy, CDN, or tunnel (nginx, Caddy, Cloudflare Tunnel) in front for any internet-facing deployment. A few behaviors only matter behind a proxy: TLS termination, the request-body size limits, Server-Sent Events buffering (WaveHouse now sends keepalive heartbeats so quiet streams survive proxy idle timeouts, [#226](https://github.com/Wave-RF/WaveHouse/issues/226)), header/auth forwarding, and which health paths to expose. See **[Behind a reverse proxy](/reverse-proxy)** for the full guide and example nginx/Caddy/Cloudflare configs.
332+
WaveHouse serves plain HTTP on `:8080` and does **not** terminate TLS, manage certificates, or rate-limit — put a reverse proxy, CDN, or tunnel (nginx, Caddy, Cloudflare Tunnel) in front for any internet-facing deployment. A few behaviors only matter behind a proxy: TLS termination, the request-body size limits, Server-Sent Events buffering (WaveHouse now sends keepalive comments so quiet streams survive proxy idle timeouts, [#226](https://github.com/Wave-RF/WaveHouse/issues/226)), header/auth forwarding, and which health paths to expose. See **[Behind a reverse proxy](/reverse-proxy)** for the full guide and example nginx/Caddy/Cloudflare configs.
333333

334334
## ClickHouse Schema
335335

0 commit comments

Comments
 (0)