Repository navigation
Closed
Closed
Copy link
Description
Why
Recordings today scale to KB-to-single-digit-MB before the architecture pushes back:
- localStorage (
bowire_recordings) — 5-10 MB origin quota. Past that, the workbench fails to save new steps. - Disk (
~/.bowire/recordings.json) — single combined JSON, loaded into memory on every read, rewritten on every write. A few MB starts feeling laggy; tens of MB blocks the UI; few GB is impossible. - In-flight memory — each recording is held fully in
recordingsList[]in the workbench. Big bodies (binary responses, large GraphQL payloads, image base64) sit in memory the whole session.
That's fine for the "capture a 6-step login flow, replay it as a test" use case. It breaks the moment an operator wants to:
- Record an MQTT subscription with 10k messages for replay analysis.
- Capture a gRPC stream that runs for an hour (large body + duration).
- Snapshot the response-set of a paginated API across thousands of pages.
- Build a fixture for an LLM context evaluation (large structured responses).
All of which are real workflows the workbench's positioning ("debug + test + mock everything multi-protocol") implies are in scope.
Proposal — chunked, lazy-loaded recording storage
Storage shape
Each recording lives in its own directory under ~/.bowire/recordings/<recordingId>/:
~/.bowire/recordings/<recordingId>/
recording.json # metadata + step-index manifest
steps/0000.json # individual step bodies, chunked
steps/0001.json
steps/...
bodies/<contentHash> # large binary bodies, content-addressed
recording.jsoncarries only the recording-level metadata + a step manifest (id, timestamp, method, byte count, body hash). Stays small (< 1 MB even for 100k-step recordings).- Step bodies live in their own JSON files. Append-only during capture; lazy-read on replay.
- Large bodies (> 1 MB threshold) get content-hashed into
bodies/so duplicate large blobs (same response repeated 50×) deduplicate naturally.
Memory shape
recordingsList[]carries only the metadata + step count, not the bodies.- Step bodies fetched on-demand when the operator opens the detail view or replay starts.
- Streaming replay reads + dispatches one step at a time, so memory stays flat regardless of total recording size.
Capture
- Each captured step writes its body to the corresponding
steps/N.jsonimmediately (append-write, fsync per step). No in-memory accumulation. - Workbench keeps a circular "last 50 steps" preview cache for the detail-pane scroll buffer.
Replay
- Replay reads steps in order from disk, decoding one at a time.
- For very-large bodies, decode-streaming directly into the target socket / channel without materializing the whole thing in JS.
Storage backends
- File system (default): the layout above.
- SQLite (optional, opt-in): single
recordings.sqlitefor operators who'd rather have one file. Same chunked-step schema. - Per-workspace store (composes with Workspaces — named, switchable contexts that bundle URLs / envs / collections / AI config #116): each workspace gets its own
recordings/directory.
Limits + UX
- A hard size cap (default 5 GB per recording, configurable via
Bowire:Recording:MaxBytes) so a runaway capture doesn't fill the disk. - The Recordings sidebar shows per-recording size + step count.
- A "Trim recording" action lets the operator drop steps before a chosen index (e.g. discard the warm-up calls).
Acceptance
-
~/.bowire/recordings/<id>/recording.json+steps/N.jsonlayout. - Append-write per step during capture (no in-memory batch).
- Body content-addressing for de-duplication.
- Lazy step-fetch in the detail view.
- Streaming replay reads one step at a time.
- Max-bytes hard cap with operator-visible warning.
- Per-recording size + step count in the sidebar.
- localStorage continues to hold metadata only; bodies never go there.
- Existing single-JSON recordings auto-migrate on first launch.
- Optional SQLite backend behind a config switch.
Composes with
- Workspaces — named, switchable contexts that bundle URLs / envs / collections / AI config #116 Workspaces — recording storage scopes per-workspace.
- Recordings: one-click 'Use as mock' spins up a local mock server #94 Use as Mock — large-recording mocks stream-read from the chunked layout instead of pre-loading.
- Parallel sessions for recordings / collections (phase 1 local, phase 2 distributed) #132 Parallel sessions — long-running parallel captures hit the chunked layout sooner than single-session capture does.
- Self-telemetry + Grafana dashboards #29 OpenTelemetry — emit recording-size + capture-rate metrics so operators see the impact in their existing observability.
Out of scope
- Cloud / S3 / blob storage backends — local-first is the foundation; remote backends as a separate plugin once the seam exists.
- Compression — body content typically compresses well, but adding it changes the on-disk format; deserve a separate decision.
- Recording-of-recording (meta-capture) — too narrow.
Metadata
Metadata
Assignees
Labels
No labels
Type
Projects
- StatusShow more project fieldsDone
Activity
feat(workbench): #116 Phase 2 — per-workspace state isolation via wsK…
feat(server): #144 Phase 1 — ChunkedRecordingStore with per-recording…
feat(server): #144 Phase 1.5 + 1.6 — per-step endpoints + per-workspa…
fix + feat: vars-resolver fragments wired + env-row indexing + #144 P…
feat(workbench): per-workspace recording-storage mode (Disk+cache vs …
feat: #144 Phase 1.8 — disk-only storage mode + manifest-only loading…
feat(workbench): #212 Phase 0 — workspace.storage as the single stora…
thomas-stegemann commentedon Jun 17, 2026
Phase 1 closeout sits in #220. This ticket continues to track Phase 2+ (filesystem watch + reconcile UI + lazy step-load on UI scroll) — see the v2.1 milestone.
thomas-stegemann commentedon Jun 24, 2026
Erledigt — Audit-Sweep 2026-06-24. Chunked Disk-Layout mit Lazy-Load + Content-Addressing in ChunkedRecordingStore.cs (Layout Z. 14-32; MaxBytesPerRecording-Cap Z. 147, Enforcement Z. 375-379/602-606).
test(coverage): wave 1 — bring 6 v2.1 endpoint suites to ≥80% line co…