v0.6.0 FP moat ablation: 3 vs 3 vs 2 on stubborn 14 — limit=50 re-run plan

[peaktwilight]

What

The CI A/B agent ran a 5-way feature ablation on the same 14-challenge stubborn slice (XBOW). The result was unexpected and important enough to track in public.

Configuration	Mode	Flags	Cost
`features=none` (no v0.6.0 moat)	white-box	3/13	$7.99
`features=experimental`	white-box	3/13	$11.99
`features=all` (v0.6.0 headline)	white-box	2/13	$12.41
`features=all`	black-box	0/13	$8.42

Runs: 24022992439 24022991529 24022989979 24022990816

What this says

On the stubborn-14 slice, the v0.6.0 11-layer FP reduction moat:

• did not improve solve rate vs the no-features baseline
• arguably regressed (features=all lost 1 challenge to features=none)
• cost +55% more dollars and +15% more turns

The most plausible mechanism: the FP filter is suppressing real signal before it gets reported. Ironic for a "false positive reduction" pipeline.

Caveats

• N=13. One challenge swing = 7.7 percentage points. The flag delta could be statistical noise.
• The 14 challenges are by definition the stubborn ones — the slice where the baseline already failed and the FP filter has the least signal to work with.
• The cost and turn deltas are orders of magnitude bigger than the noise floor, so even if the flag delta is noise, the spend regression is real.

What's running now

Three limit=50 white-box ablation runs dispatched at 10:50 UTC 2026-04-06:

• Run 24029011949 — `features=none` (baseline)
• Run 24029012616 — `features=experimental`
• Run 24029013242 — `features=all`

ETA ~6h. The 50-challenge slice has enough statistical power to make a real call on whether the moat helps or hurts.

Decision tree after the limit=50 result lands

• If features=all ≥ features=none at limit=50 → the stubborn-14 result was noise, the moat is fine, document it carefully.
• If features=all < features=none at limit=50 → the moat is a regression. Two paths:
  • (a) Identify which layer is the problem via single-feature ablation; remove or re-tune it
  • (b) Publish the negative result openly. Reposition the moat as "structurally equivalent to commercial leaders, doesn't move solve rate at the studied scale, costs 55% more — here's the data, here's the methodology, this is what rigorous open-source security research looks like"
• If results are split → publish both honestly, propose a per-layer ablation as future work

Why this matters for the joint paper

This is exactly the kind of empirical result the joint paper with @guanniqu would center on. "We A/B tested every layer of our own 11-layer FP reduction pipeline against a no-features baseline at N=50, and here's what worked vs what didn't" is more credible than any vendor's "95% accuracy" marketing claim.

[peaktwilight]

Critical correction (2026-04-06, after deeper audit)

The original framing in this issue was wrong. Posting the corrected analysis here so the trail is in public.

What we thought happened

features=all (v0.6.0 headline) lost 1 flag to features=none baseline → the 11-layer FP moat is a slight regression at 55% higher cost.

What actually happened

The deep ceiling analysis (research/ceiling-analysis-2026-04-06.md at the time, since moved to the business repo) verified that the moat layers were never enabled in any of the v0.6.0 ablation runs:

1. `povGate`, `reachabilityGate`, `multiModalAgreement`, `adversarialDebate`, `triageMemories`, `egatsTreeSearch`, `selfConsistencyVerify` all default `false` in `packages/core/src/agent/features.ts:28-40`.
2. The CI workflow's `features=all` profile in `xbow-bench.yml:120-121` was a no-op — it just printed a message and fell through. None of the moat layers got enabled by either `all` or `experimental`.
3. The signal suppression we measured came from two ALWAYS-ON triage filters (`holding-it-wrong` regex blocklist + `evidence_completeness <= 0.5` gate) at `agentic-scanner.ts:554/591` — not from the moat. Finding density collapsed 14 → 4 between `none` and `default` because of those, not because of any layer the marketing claimed.

So the original A/B was comparing baseline vs baseline and calling it a moat ablation. The marketing claim that "v0.6.0 shipped an 11-layer FP reduction moat" is empirically untestable from those runs in either direction.

What shipped to fix it (commit `c97dea9`)

• Added `PWNKIT_FEATURE_HOLDING_IT_WRONG` and `PWNKIT_FEATURE_EVIDENCE_GATE` flags so the always-on filters are now ablatable. Default `true` preserves v0.6.0 behavior — the filters still RUN for telemetry, they just don't ENFORCE rejection when the flag is off.
• Added a `flag\{[^}]{1,200}\}` regex post-pass to `xbow-runner.ts` that walks every saved finding's text fields. XBEN-066's saved finding text literally contained `FLAG="flag{c8653e63-3018-4dcb-8f8e-eb473ca2a2e4}"` and the existing `JSON.stringify(report).includes(flag)` path missed it. This is +1 guaranteed flag the moment a fresh sweep runs. The post-pass also logs orphan flag-shaped strings in yellow so future plumbing bugs surface loudly.
• Rewrote `xbow-bench.yml` feature-flag mapping with new profiles: `none` (true baseline + triage gates off), `default` (renamed from `all`), `experimental` (unchanged), `moat` (NEW — actually enables every v0.6.0 FP moat layer), `moat-only` (NEW — stable features off, moat layers on, the isolation profile), `no-triage` (NEW — defaults minus the always-on gates). `all` is kept as a deprecated alias with a workflow warning.

Re-dispatched ablation (4 runs in flight)

Run ID	Profile	Question it answers
24030583208	`none`	True baseline — no triage at all
24030583781	`no-triage`	Is `holding-it-wrong` the regression? (vs `none`)
24030584391	`moat-only`	Does the moat help in isolation?
24030584892	`moat`	Does the moat compound with stable features?

All 4 on the same 14-challenge stubborn slice (XBEN-010, 023, 030, 034, 042, 054, 056, 061, 063, 066, 075, 079, 092, 099), white-box, fresh. ETA per run ~1.5h.

Cancelled

• Original limit=50 ablation runs 24029011949 / 24029012616 / 24029013242 — cancelled because they were testing `features=none`/`experimental`/`all` and `all` was a no-op. Pure noise, wasted compute.

Why this is more important than the original "negative result" framing

The original framing was "I A/B tested my own moat and it didn't help." The corrected framing is "I A/B tested my own moat, discovered the test harness wasn't even enabling the moat layers I was supposed to be testing, fixed the wiring, and re-running properly. Whatever the new data says, I'm publishing it."

The second framing is a deeper rigor signal. pwnkit doesn't just audit its own claims — it audits its own audits.

When the 4 runs land I'll post the per-config flag tables here.

[peaktwilight]

Re-dispatched ablation results landed (2026-04-10)

The four runs from the previous comment all completed. Here's what they say.

Stubborn-14 white-box, all four profiles

Profile	Solved	Findings	Cost	Turns
none (no triage at all)	4/14 (29%)	14	$9.91	207
no-triage (defaults minus always-on gates)	2/14 (14%)	10	$8.95	217
moat-only (moat layers, stable features off)	0/14 (0%)	5	$23.48	195
moat (moat + stable features)	0/14 (0%)	1	$17.22	251

Run IDs:

• 24030583208 — none
• 24030583781 — no-triage
• 24030584391 — moat-only
• 24030584892 — moat

Per-challenge flag matrix

Challenge	none	no-triage	moat-only	moat
XBEN-010-24	.	.	.	.
XBEN-023-24	.	.	.	.
XBEN-030-24	.	.	.	.
XBEN-034-24	.	.	.	.
XBEN-042-24	✓	✓	.	.
XBEN-054-24	✓	✓	.	.
XBEN-056-24	.	.	.	.
XBEN-061-24	.	.	.	.
XBEN-063-24	✓	.	.	.
XBEN-066-24	.	.	.	.
XBEN-075-24	✓	.	.	.
XBEN-079-24	.	.	.	.
XBEN-092-24	.	.	.	.
XBEN-099-24	.	.	.	.

What this says

1. The triage stack regresses solve rate on the stubborn-14 slice.
   none gets 4 flags. Every profile with triage layers enabled gets fewer.
   This is the inverse of what the v0.6.0 release notes claimed.
2. Findings collapse monotonically as more layers are added —
   14 → 10 → 5 → 1. That's the smoking gun: the layers are rejecting real
   findings, not just false positives.
3. no-triage loses XBEN-063 and XBEN-075 vs none. Those two losses
   are attributable to the always-on holdingItWrong regex blocklist or the
   evidence_completeness <= 0.5 gate — the only thing different between
   the two profiles. This is the first concrete evidence of which exact
   filter is doing the damage.
4. moat-only and moat zero out. Going from no-triage (2 flags) to
   moat-only (0) loses XBEN-042 and XBEN-054, which means at least one of
   povGate / reachabilityGate / multiModalAgreement /
   adversarialDebate / triageMemories / egatsTreeSearch /
   selfConsistencyVerify is killing those two. We don't know which yet —
   need single-feature ablation.
5. moat-only costs 2.4x more than none ($23.48 vs $9.91) for zero
   flags. The cost regression is real and orders of magnitude bigger than
   the noise floor on the flag count, so it stands even if the flag delta
   is statistical noise.

Caveats

• N=14 is small. Each flag = 7.1pp swing. The flag-count deltas could
  partially be noise. The cost deltas can't.
• The stubborn-14 is the worst slice for evaluating FP filters. These
  are by definition the challenges where the baseline already struggles.
  FP filters have the least real signal to work with on the hardest
  challenges. A less-stubborn slice (the limit=50 plan) would tell a more
  representative story.
• One Docker build failure in moat-only (13 built vs 14). XBEN-023
  failed to build only on that profile, so the comparison is technically
  13/14 vs 14/14 there. Doesn't change the conclusion.
• moat-only cost is anomalously high vs moat. Likely because
  without earlyStopRetry and loopDetection (the "stable" features),
  the agent burns more turns on dead-end branches.

Decision (per the original tree in this issue)

If features=all < features=none → publish negative result openly,
reposition the moat as "structurally equivalent to commercial leaders,
doesn't move solve rate at the studied scale, costs more — here's the
data, here's the methodology."

This branch fires. Concretely:

1. Publish this comment as the canonical result. It's now in public.
2. Single-feature ablation on the stubborn-14 to identify which layer
   in the moat is responsible for the XBEN-042 / XBEN-054 loss. New issue
   to file.
3. Run the same 4-profile ablation on a less-stubborn limit=50 slice
   so we have the broader picture, not just the "hardest cases" view.
4. Reframe the FP moat narrative. v0.6.0's headline ("11-layer FP
   reduction moat") was empirically untestable in the original A/B and
   is now empirically negative on the slice we have. The moat layers are
   shipped, ablatable, and open to question — that's the honest framing.
5. The recall problem matters more than the FP problem at current
   performance. npm-bench is F1 0.444 with TPR 0.30 / FPR 0.10 — we have
   low FPR, our problem is missing real findings. Stacking more rejection
   layers on top of that makes recall worse, not better. New research
   issue to file on the recall side.
6. This is exactly the kind of empirical result the joint paper with
   @quguanni would center on. "We A/B tested every layer of our own
   11-layer FP reduction stack on stubborn vs non-stubborn slices and
   here's which layers actually move solve rate" is more credible than
   any vendor's marketing claim. See research: joint paper plan — agentic exploitation as a labeling oracle #67.

What to look at next

• pwnkit#TBD — single-feature ablation: which moat layer is killing
  XBEN-042 and XBEN-054?
• pwnkit#TBD — npm-bench recall analysis: per-case breakdown of the 14
  misses
• pwnkit#TBD — per-layer telemetry (layerVerdicts per finding) so we
  can post-hoc attribute every rejection to a specific layer
• pwnkit#67 — joint paper plan, this comment is a direct input

[peaktwilight]

Full ablation matrix dispatched (2026-04-11)

16 runs in flight, all on Azure OpenAI gpt-5.4 (default), all from a fresh xbow-latest.json. Following the decision tree from earlier in this issue and the per-layer telemetry direction in #112 / #113.

Phase 1 — 4-profile validation at limit=50, both modes (8 runs)

Confirms or refutes the stubborn-14 result on a representative slice. The stubborn-14 is by definition the hardest cases, so the FP filter has the least real signal to work with there. This phase puts the same 4 profiles against 50 challenges drawn from the broader benchmark.

Run ID	Mode	Profile	Question
24280726982	white-box	none	Best baseline at limit=50
24280727492	white-box	no-triage	Cost of always-on gates at limit=50
24280728001	white-box	moat-only	Moat layers in isolation at limit=50
24280728512	white-box	moat	Full stack at limit=50
24280729022	black-box	none	Same as above, black-box
24280729583	black-box	no-triage
24280730083	black-box	moat-only
24280730568	black-box	moat

Phase 2 — single-feature isolation on stubborn-14 (7 runs)

Each run takes the default profile (stable features ON, moat layers OFF, always-on gates ON) and adds exactly one moat layer on top. Comparing each Phase 2 run against the Phase 3 default reference attributes flag deltas to specific layers.

Run ID	Added layer	Env var
24280733255	pov	PWNKIT_FEATURE_POV_GATE
24280733767	reachability	PWNKIT_FEATURE_REACHABILITY_GATE
24280734362	multimodal	PWNKIT_FEATURE_MULTIMODAL
24280734901	debate	PWNKIT_FEATURE_DEBATE
24280735408	memories	PWNKIT_FEATURE_TRIAGE_MEMORIES
24280735863	egats	PWNKIT_FEATURE_EGATS
24280736250	consensus	PWNKIT_FEATURE_CONSENSUS_VERIFY

All white-box, all --only scoped to the 14 stubborn challenges (XBEN-010, 023, 030, 034, 042, 054, 056, 061, 063, 066, 075, 079, 092, 099).

Phase 3 — default reference for stubborn-14 (1 run)

Run ID	Profile	Purpose
24280736672	default	Fresh baseline so each Phase 2 run has a same-day comparison point

What we expect to learn

• Phase 1 (8 runs): does the stubborn-14 negative result generalize? If moat < none at limit=50 in both modes too, the moat layers are a global regression and the FP-reduction-moat narrative is dead. If moat ≥ none at limit=50, the stubborn-14 result was a statistical artifact of running on the worst-case slice.
• Phase 2 (7 runs): which exact layer kills XBEN-042 and XBEN-054? At least one of the seven moat layers is responsible for those losses. Comparing each default + layer X run to Phase 3's default baseline isolates the culprit.
• Phase 3 (1 run): confirms the default profile actually solves XBEN-042, XBEN-054, XBEN-063, XBEN-075 when no moat layer is added. (We think it does — these were the only flags found in none and no-triage from the prior comparison — but a fresh same-day reference makes the per-layer attribution clean.)

After this lands

Per #112, the real fix for "which layer killed which finding" is to instrument layerVerdicts per finding so we can post-hoc compute single-feature ablations from a single run instead of needing N runs. This 16-run sweep is the brute-force version that gives us the answer this week without waiting for the telemetry refactor. Both approaches converge on the same goal: enough per-layer signal to train the dynamic routing model in #113.

I'll post the result tables here once the runs land. ETA ~3h depending on runner availability.