[research] Dynamic triage routing: learn per-finding layer selection (paper-novel contribution)

[peaktwilight]

What

Train a small classifier that decides per finding which subset
of pwnkit's 11 triage layers to invoke, instead of statically
toggling them via env vars. The training signal is the per-layer
telemetry from #112.

This is the novel contribution worth publishing in the joint paper
with @quguanni (#67) — every existing hybrid system uses a static
pipeline. Nobody has shipped learned routing.

Motivation

Per #72's per-profile ablation:

Profile	Solved	Findings	Cost
none (no triage at all)	4/14	14	$9.91
no-triage	2/14	10	$8.95
moat-only	0/14	5	$23.48
moat	0/14	1	$17.22

The triage layers help on some findings and hurt on others. Static
toggling at the scan level is the wrong granularity:

• For a high-confidence SQLi with a clear error-based signal, the
  oracle is free, deterministic, and sufficient — running the
  expensive adversarialDebate layer adds latency, cost, and
  removes a real finding (per the ablation).
• For an ambiguous logic bug where the agent's confidence is 0.4,
  the structured 4-step verify and the PoV gate add real signal.
• For a finding matching a known FP pattern in triageMemories,
  every other layer should be skipped — auto-reject.

The right granularity is per-finding, not per-scan. The routing
decision is itself a learnable function of the finding's features
and category.

Architecture

Finding arrives
    ↓
Layer 1 feature extractor (already shipped)
    ↓
45-dim vector + finding text
    ↓
Routing classifier (NEW)
    ↓
{
  "tp_score": 0.72,
  "layers_to_run": ["oracle", "structured_verify"],
  "layers_to_skip": ["pov_gate", "adversarial_debate", "memories"],
  "auto_accept_if_score_above": 0.95,
  "auto_reject_if_score_below": 0.20
}
    ↓
For each layer in layers_to_run: invoke
For each layer in layers_to_skip: don't invoke
Aggregate verdicts → final triage decision

The routing head can be a multi-label classifier sitting alongside
the TP/FP head, both consuming the same 45-feature vector + text
embedding from the VulnBERT-style hybrid encoder.

Training data

Comes from #112 once it lands. Each finding becomes a row of:

{
  "features": [...45 numbers],
  "text": "Title: ...\nCategory: ...\n...",
  "ground_truth_label": 1,
  "layer_verdicts": [
    { "layer": "holding_it_wrong", "verdict": "pass", "duration_ms": 0.3 },
    { "layer": "evidence_gate", "verdict": "pass", "duration_ms": 0.1 },
    { "layer": "oracle", "verdict": "pass", "confidence": 0.95, "duration_ms": 4200 },
    { "layer": "structured_verify", "verdict": "pass", "confidence": 0.88, "duration_ms": 18400, "cost_usd": 0.12 },
    { "layer": "adversarial_debate", "verdict": "reject", "confidence": 0.62, "duration_ms": 31200, "cost_usd": 0.41 }
  ],
  "final_verdict": "rejected"
}

The router's learning objective: predict which subset of layers
would have produced the same final verdict at the lowest cost.
A layer that never changes the verdict for findings of a given
shape is dead weight for that shape — the router learns to skip it.

Why this is novel

Existing hybrid vulnerability triage systems use static pipelines:

System	Architecture	Routing
Datadog Bits AI	LLM classifier → SAST findings	Static (one model on every finding)
GitHub Sec Lab Taskflow	GPT-4 + 7 YAML subtasks	Static (every subtask runs every time)
Endor Labs AI SAST	Rules + reachability + LLM	Static (rules → reachability → LLM, fixed order)
Semgrep Assistant	LLM + per-target memories	Static (memory check → LLM, fixed order)
SAST-Genius (arxiv 2509.15433)	Static analysis + LLM hybrid	Static
SecureFixAgent (arxiv 2509.16275)	Bandit + LoRA LLM iterative loop	Static
VulnBERT (Pebblebed)	CodeBERT + features + cross-attention	Static (one model, one verdict)

Nobody routes findings to different verification stages based on a
learned function of the finding's content. That's the gap.

The pwnkit-side analogy

pwnkit already has a precedent for this pattern, just not at the
triage layer:

• payload_lookup — the agent doesn't get the 22KB JSFuck XSS
  payload in its context by default. It calls payload_lookup when
  it detects a letter/digit filter. The agent decides when to
  unlock the capability.
• dynamicPlaybooks — vulnerability-specific playbooks (SQLi,
  SSTI, auth chain) get auto-injected at ~30% of the budget based
  on what the agent observed during recon. The injection is
  conditional on detected patterns, not static.
• EGATS — the agent expands attack-tree branches based on
  evidence scores, not a fixed schedule.

All three are existing dynamic-capability-unlock patterns. The
routing classifier is the same idea applied to the triage stage:
unlock layers based on what the finding looks like, not on what
the operator set in env vars.

What this issue is asking for

Phase 1 — design doc only. No code yet. Specifically:

• Pick the routing model class (multi-label LR over the 45
  features? small MLP head on the VulnBERT-web encoder? per-category
  gating tree?). Defend the choice with the constraint that
  inference must be sub-millisecond per finding.
• Pick the training objective. Multi-label binary cross-entropy
  over (layer, would_produce_same_final_verdict) is the
  obvious starting point but needs to be defended against
  simpler baselines.
• Define the evaluation metric. Cost-saved-per-recall-lost is
  the right shape but the units need pinning down.
• Define the rollout: feature flag (PWNKIT_FEATURE_LEARNED_ROUTER),
  A/B harness against the existing static stack, the comparison
  points (none, default, moat, router).
• Identify what training data we need beyond [research] Per-layer triage telemetry: log layerVerdicts on every finding #112's telemetry
  and how much of it (are 100 labeled findings enough? 10k?)

Phase 2 (separate issue, blocked on this) — actually train a model
and run the A/B.

Out of scope

• Picking the text encoder (CodeBERT vs Llama-8B vs ?). Guanni's
  call (research: joint paper plan — agentic exploitation as a labeling oracle #67).
• Replacing the existing 11 layers with new ones. This issue is
  about routing the existing layers, not redesigning them.
• Production-grade ONNX runtime integration. Phase 2.

Related

• research: joint paper plan — agentic exploitation as a labeling oracle #67 — joint paper plan, this is the novel contribution
• v0.6.0 FP moat ablation: 3 vs 3 vs 2 on stubborn 14 — limit=50 re-run plan #72 — ablation that motivated this (showed static layers help
  on some findings, hurt on others)
• [research] npm-bench recall analysis: per-case breakdown of the 14 misses #111 — npm-bench recall analysis (the target metric to improve)
• [research] Per-layer triage telemetry: log layerVerdicts on every finding #112 — per-layer telemetry (hard prerequisite, must land first)

[peaktwilight]

Ablation data (2026-04-11) now validates this issue

The 21-run ablation matrix posted in #72 provides the empirical motivation this issue was missing. Three key findings, all of which point at learned per-finding routing as the right answer:

1. No single static policy wins on all three benchmark slices

Slice	Best profile	Worst profile	Notes
XBOW white-box @ limit=50	no-triage (44/50)	moat (41/50)	moat costs 2 flags, 63% fewer findings, 1.6× $/flag
XBOW black-box @ limit=25	moat (19/25, $0.53/flag)	none (18/25, $0.76/flag)	moat strictly dominates — more flags, fewer findings, cheaper
npm-bench (81 pkgs)	none (F1 0.973)	moat/default (F1 0.956)	moat is a no-op; stable features are the FPR offender

A learned per-finding router would pick no-triage behavior on white-box XBOW, moat behavior on black-box XBOW, and none behavior on npm-bench — per finding, not per scan. That's exactly the contribution this issue was proposed to deliver.

2. Single-feature isolation shows different layers help different findings

From the stubborn-14 single-feature ablation:

Layer added to default	Flags vs default	Cost
reachability	+3	$8.04
debate	+3	$13.26
pov	+2	$9.56
memories	+2	$13.40
consensus	+1	$8.01
multimodal	+1	$7.55
egats	−1	$15.93

reachability wins on cost-per-flag at $1.61. debate matches on flag count but at 8× the cost. A router that picked reachability on cheap-to-reach-but-hard-to-exploit findings and debate on ambiguous-evidence findings would dominate any static policy on this slice.

3. Per-finding telemetry is now instrumented (#112 shipped today)

Commit 6f1a889 adds Finding.layerVerdicts: LayerVerdict[] — each finding now logs which triage layer touched it, what verdict the layer returned, confidence, duration, cost, and any severity transition. 6 of 10 triage layers instrumented (holding_it_wrong, evidence_gate, reachability, multi_modal, oracle, pov_gate).

That's the supervision signal this issue's training data needs. Every (features, text, layer_outcomes, final_verdict) tuple is now recoverable from a single benchmark run instead of needing N runs.

4. First training dataset committed today

Commit f40e1c1 ships packages/benchmark/results/triage-dataset-v1.jsonl — 969 labeled rows from the 21 ablation runs. 884 TP / 85 FP (imbalanced, expected for benchmark-sourced data), stratifiable by profile via the source field.

v1 is mostly empty layer_verdicts arrays because most rows predate #112. v2 will have layerVerdicts across the board once a fresh run lands against the post-6f1a889 commit. That's the version that can actually train a router.

Updated scope for this issue

The design phase from the original issue is still the right shape (multi-label routing head alongside the TP/FP classifier, sub-millisecond inference, cost-saved-per-recall-lost metric). What the ablation data changes:

1. The training task is cleaner than I wrote originally. Rather than learning "which subset of N layers is optimal?", the router can be trained to learn "which single dominant signal does this finding need?" — reachability alone, debate alone, oracle alone, or skip-all. The stubborn-14 data shows those four options cover most of the variance.

2. The routing inputs should include slice-level features — mode (wb/bb), target type (npm/web/source), finding category. The ablation shows these are first-order predictors of which layer helps. A router that doesn't see them will fail to learn the per-slice specialization.

3. The baseline to beat is not default — it's the best static profile per slice. That's no-triage on white-box XBOW, moat on black-box XBOW, none on npm-bench. The router has to beat all three on their own turf simultaneously.

4. Phase 1 is still design-doc-only. But the ablation data makes the design decisions much easier — we already know the answer to most of the open questions.

Next step on this issue: write the design doc, commit it as a research page in docs, open a follow-up issue for the training code.

Related

• #72 ablation — the data that motivates this
• #112 — per-finding layer telemetry (prerequisite, now shipped)
• #114 — triage-dataset-v1 (first training data, now shipped)
• #116 — egats disable (now done; routing model should still learn when egats helps, but the default no longer pays its cost)
• #67 — joint paper plan with @quguanni (this issue is the novel paper contribution)

[peaktwilight]

Phase 3 baseline: XGBoost on 45-feature vector (2026-04-12)

Trained on triage-dataset-v2.jsonl (1514 rows, 1410 TP / 104 FP). Model saved at packages/benchmark/results/triage-router-v1.json.

Aggregate 5-fold CV

Metric	Score
F1	0.944 ± 0.008
Precision	0.969 ± 0.006
Recall	0.920 ± 0.013
AUC	0.886 ± 0.028

Leave-one-slice-out (the real test)

Held-out slice	F1	TP recall	FP recall	Verdict
npm-bench (901 samples)	0.664	50%	84%	fails — xbow features don't transfer to npm
xbow-bb (115 samples)	0.859	78%	33%	marginal — loses 22% of TPs
xbow-wb (498 samples)	0.900	93%	12%	OK — but misses 88% of FPs

The cross-slice generalization is poor. A model trained on xbow catches only 50% of npm-bench TPs. The aggregate 0.944 F1 is misleading because it's trained and tested on mixed data.

Within-slice 5-fold (same-domain test)

Slice	F1
npm-bench	0.930 ± 0.025
xbow-wb	0.914 ± 0.023
xbow-bb	0.721 ± 0.363

Within-domain performance is strong for npm-bench and xbow-wb. xbow-bb has too few samples (n=115) for reliable CV.

Feature importance is completely different per slice

Rank	npm-bench	xbow-bb	xbow-wb
#1	text_description_length (49.6%)	cross_resp_req_length_ratio (52.5%)	req_path_traversal (12.3%)
#2	text_analysis_length (11.6%)	meta_injection_class (19.2%)	req_param_count (12.2%)
#3	cross_resp_req_length_ratio (11.6%)	resp_length (17.4%)	resp_error_message (7.0%)
#4	req_encoding_detected (10.0%)	text_description_length (8.9%)	req_command_injection (5.7%)
#5	text_impact_statement (5.0%)	meta_high_confidence_category (2.0%)	resp_redirect (5.6%)

This is the strongest evidence yet for per-finding routing. The three slices use completely different signal. npm-bench is basically "longer descriptions = more likely TP." xbow-bb is "bigger response/request ratio = more likely TP." xbow-wb actually looks at exploit indicators. A single mixed classifier can't optimize all three simultaneously.

What this means for the design doc

Per the original Phase 4 decision tree:

If Option A clears the bar, ship it. If it plateaus, proceed to Phase 5.

Option A (XGBoost on features alone) does NOT clear the bar for cross-slice generalization. 0.664 F1 on npm-bench when trained on xbow is well below the bar of "beat every static profile on its home turf."

Two paths forward:

Path A (recommended): add slice-type as a feature. The design doc already listed "per-scan inputs: mode (wb/bb), target type (npm/web/source)" as first-class features. Adding a one-hot slice-type feature to the 45-dim vector is trivial and directly addresses the cross-slice failure. This is still Option A (XGBoost), just with the per-scan context the design doc said was needed.

Path B: per-slice classifiers. Train 3 separate XGBoost models (one per slice), dispatch to the right one based on target type. Simpler, no cross-contamination, but doesn't learn transfer between slices.

I'll try Path A next — add mode + target-type features, retrain, re-evaluate.