You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Benchmarks: k6-style thresholds as CI gates — fail bowire test / bench runs on latency budgets #360
Competitive-research input (mid-2026): k6's thresholds (p95 < 200ms → non-zero exit) are the load-testing table stake, and k6 2.0 doubled down (OTel output, threshold gates unchanged at the core). Bowire has the benchmark envelope + exports (#234) and a CI runner (#181) but no way to make a pipeline fail on a latency regression.
Proposal — same DSL surface as flow expectations, so operators learn one thing:
Threshold spec on the benchmark envelope: metric (p50/p95/p99/error-rate/throughput) + operator + budget
bowire bench run --fail-on-threshold → exit non-zero when any threshold breaks; TTY summary marks the breaching metric
Thresholds land in the existing k6-summary JSON export so downstream tooling agrees
The design decision that mattered: no second implementation. The measurement had lived entirely in the rail's JavaScript, so the obvious route — a fresh CLI benchmark loop — would have produced a second set of numbers. Instead:
the runner drives IBowireProtocol.InvokeAsync, the same path bowire call and bowire test take, so a benchmark measures the request path the rest of the tool actually uses;
percentiles use nearest-rank (ceil(p/100 × n) − 1), which is what the rail's _percentileSorted has always used;
success/failure follows isHistoryEntryOk from the workbench verbatim.
A p95 read in the rail is the p95 CI compares against. Had those drifted, the feature would have been actively misleading rather than merely incomplete.
Where the pieces live.BowireBenchmarkStats, BowireBenchmarkThreshold, BowireK6Summary and BowireBenchmarkRunner sit in Kuestenlogik.Bowire.Benchmarking — which references only Core, so the CLI reaches them today and the MCP tool can reach the same engine later without a third code path.
Thresholds.metric operator budget over p50/p90/p95/p99/avg/min/max/error-rate/throughput. k6's own p(95) spelling parses, as do error_rate / rps / median, so a budget can be copied out of a k6 script. Parsing is invariant (0.01 means the same on a German-locale runner) and happens before the run starts — a typo in the last of five budgets should not surface after a two-minute load test.
k6 export.--k6-summary writes the #234 metric block with each budget attached the way k6 reports its own — keyed by source text inside the metric it constrains:
so a dashboard already ingesting k6 summaries finds Bowire's budgets where it looks for k6's.
Verified end-to-end against a live mock, which is also how a display bug surfaced: throughput was being rendered through the millisecond formatter and printed 1315ms req/s. Now 59.2 req/s. The gate itself: a deliberately impossible budget prints FAIL p95<0.001 (actual 3.96ms) and exits 1; a satisfied one passes and exits 0.
Tests: 56, in a new Kuestenlogik.Bowire.Benchmarking.Tests project — the package shipped without one, which is part of what #358 tracks. They cover the nearest-rank boundaries against hand-computed values, threshold parsing and rejection, strict-vs-inclusive at the exact budget (the difference between a green and a red pipeline), k6's failure-rate inversion, and the runner's iteration count, concurrency overlap, warm-up and error paths against a scripted protocol.
Docs:performance.md gains the CLI section, the flag table, the k6 threshold shape, and the split the ticket asks to document — a flow expectation (kind: latency) is a per-request functional budget, a benchmark threshold is an aggregate distribution budget. It also corrects a stale claim that runs cannot be exported.
Activity
feat(bench): threshold engine, stats and k6 summary for CI gates (#360)
feat(bench): `bowire bench run` with threshold gates and k6 export (#360
thomas-stegemann commentedon Aug 19, 2026
Shipped. Benchmarks stop being informational — a pipeline can now fail on a latency regression.
The design decision that mattered: no second implementation. The measurement had lived entirely in the rail's JavaScript, so the obvious route — a fresh CLI benchmark loop — would have produced a second set of numbers. Instead:
IBowireProtocol.InvokeAsync, the same pathbowire callandbowire testtake, so a benchmark measures the request path the rest of the tool actually uses;ceil(p/100 × n) − 1), which is what the rail's_percentileSortedhas always used;isHistoryEntryOkfrom the workbench verbatim.A p95 read in the rail is the p95 CI compares against. Had those drifted, the feature would have been actively misleading rather than merely incomplete.
Where the pieces live.
BowireBenchmarkStats,BowireBenchmarkThreshold,BowireK6SummaryandBowireBenchmarkRunnersit inKuestenlogik.Bowire.Benchmarking— which references only Core, so the CLI reaches them today and the MCP tool can reach the same engine later without a third code path.Thresholds.
metric operator budgetoverp50/p90/p95/p99/avg/min/max/error-rate/throughput. k6's ownp(95)spelling parses, as doerror_rate/rps/median, so a budget can be copied out of a k6 script. Parsing is invariant (0.01means the same on a German-locale runner) and happens before the run starts — a typo in the last of five budgets should not surface after a two-minute load test.k6 export.
--k6-summarywrites the #234 metric block with each budget attached the way k6 reports its own — keyed by source text inside the metric it constrains:so a dashboard already ingesting k6 summaries finds Bowire's budgets where it looks for k6's.
Verified end-to-end against a live mock, which is also how a display bug surfaced: throughput was being rendered through the millisecond formatter and printed
1315ms req/s. Now59.2 req/s. The gate itself: a deliberately impossible budget printsFAIL p95<0.001 (actual 3.96ms)and exits 1; a satisfied one passes and exits 0.Tests: 56, in a new
Kuestenlogik.Bowire.Benchmarking.Testsproject — the package shipped without one, which is part of what #358 tracks. They cover the nearest-rank boundaries against hand-computed values, threshold parsing and rejection, strict-vs-inclusive at the exact budget (the difference between a green and a red pipeline), k6's failure-rate inversion, and the runner's iteration count, concurrency overlap, warm-up and error paths against a scripted protocol.Docs:
performance.mdgains the CLI section, the flag table, the k6 threshold shape, and the split the ticket asks to document — a flow expectation (kind: latency) is a per-request functional budget, a benchmark threshold is an aggregate distribution budget. It also corrects a stale claim that runs cannot be exported.Commits: 7065467 (engine + tests), 29a8fb0 (CLI + docs). CI green.
Follow-up: #232 (scheduled runs + cron infrastructure) now has a CLI runner to schedule.
docs(release-notes): reset upcoming.md to 2.5 and move v2.4's breakin…