Skip to content

Benchmarks: k6-style thresholds as CI gates — fail bowire test / bench runs on latency budgets #360

Closed

Description

@thomas-stegemann

Competitive-research input (mid-2026): k6's thresholds (p95 < 200ms → non-zero exit) are the load-testing table stake, and k6 2.0 doubled down (OTel output, threshold gates unchanged at the core). Bowire has the benchmark envelope + exports (#234) and a CI runner (#181) but no way to make a pipeline fail on a latency regression.

Proposal — same DSL surface as flow expectations, so operators learn one thing:

  • Threshold spec on the benchmark envelope: metric (p50/p95/p99/error-rate/throughput) + operator + budget
  • bowire bench run --fail-on-threshold → exit non-zero when any threshold breaks; TTY summary marks the breaching metric
  • Thresholds land in the existing k6-summary JSON export so downstream tooling agrees
  • Flow steps already assert per-request latency (kind: latency) — document the split: per-request budget = flow expectation, aggregate budget = benchmark threshold

This closes the 'benchmarks are informational only' gap vs k6 while keeping everything in one tool — no second DSL, no second runner.

Activity

  1. thomas-stegemann commented on Aug 19, 2026

    @thomas-stegemann
    MemberAuthor

    Shipped. Benchmarks stop being informational — a pipeline can now fail on a latency regression.

    bowire bench run Weather/getCurrent -url rest@http://localhost:6000 \
      -n 500 -c 8 --warmup 20 \
      --threshold "p95 < 200" --threshold "error-rate < 0.01" \
      --fail-on-threshold

    The design decision that mattered: no second implementation. The measurement had lived entirely in the rail's JavaScript, so the obvious route — a fresh CLI benchmark loop — would have produced a second set of numbers. Instead:

    • the runner drives IBowireProtocol.InvokeAsync, the same path bowire call and bowire test take, so a benchmark measures the request path the rest of the tool actually uses;
    • percentiles use nearest-rank (ceil(p/100 × n) − 1), which is what the rail's _percentileSorted has always used;
    • success/failure follows isHistoryEntryOk from the workbench verbatim.

    A p95 read in the rail is the p95 CI compares against. Had those drifted, the feature would have been actively misleading rather than merely incomplete.

    Where the pieces live. BowireBenchmarkStats, BowireBenchmarkThreshold, BowireK6Summary and BowireBenchmarkRunner sit in Kuestenlogik.Bowire.Benchmarking — which references only Core, so the CLI reaches them today and the MCP tool can reach the same engine later without a third code path.

    Thresholds. metric operator budget over p50/p90/p95/p99/avg/min/max/error-rate/throughput. k6's own p(95) spelling parses, as do error_rate / rps / median, so a budget can be copied out of a k6 script. Parsing is invariant (0.01 means the same on a German-locale runner) and happens before the run starts — a typo in the last of five budgets should not surface after a two-minute load test.

    k6 export. --k6-summary writes the #234 metric block with each budget attached the way k6 reports its own — keyed by source text inside the metric it constrains:

    "http_req_duration": {
      "values": { "p(95)": 3.9571 },
      "thresholds": { "p95<0.001": { "ok": false } }
    }

    so a dashboard already ingesting k6 summaries finds Bowire's budgets where it looks for k6's.

    Verified end-to-end against a live mock, which is also how a display bug surfaced: throughput was being rendered through the millisecond formatter and printed 1315ms req/s. Now 59.2 req/s. The gate itself: a deliberately impossible budget prints FAIL p95<0.001 (actual 3.96ms) and exits 1; a satisfied one passes and exits 0.

    Tests: 56, in a new Kuestenlogik.Bowire.Benchmarking.Tests project — the package shipped without one, which is part of what #358 tracks. They cover the nearest-rank boundaries against hand-computed values, threshold parsing and rejection, strict-vs-inclusive at the exact budget (the difference between a green and a red pipeline), k6's failure-rate inversion, and the runner's iteration count, concurrency overlap, warm-up and error paths against a scripted protocol.

    Docs: performance.md gains the CLI section, the flag table, the k6 threshold shape, and the split the ticket asks to document — a flow expectation (kind: latency) is a per-request functional budget, a benchmark threshold is an aggregate distribution budget. It also corrects a stale claim that runs cannot be exported.

    Commits: 7065467 (engine + tests), 29a8fb0 (CLI + docs). CI green.

    Follow-up: #232 (scheduled runs + cron infrastructure) now has a CLI runner to schedule.

  2. moved this from Backlog to Done in Bowireon Aug 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    • Status
      Done

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions