research: joint paper plan — agentic exploitation as a labeling oracle

[peaktwilight]

What

Tracking issue for the joint research paper between Doruk (pwnkit) and Guanni Qu (VulnBERT, Pebblebed Research Resident). Working title:

Agentic exploitation as a labeling oracle for vulnerability triage models

The thesis

The training data for a high-precision vulnerability triage classifier only exists if you have BOTH:

1. an agentic exploit harness running at scale (pwnkit's role) that produces `(finding, attempted-PoC, real-or-fake)` tuples
2. a labeled-classifier training pipeline (VulnBERT-class hybrid features-plus-embeddings architecture) that can consume those tuples and produce a small specialized model

Neither half exists meaningfully without the other. The dataset itself — not the model — is the moat. This is the inverse of the standard "dataset is given, model is the asset" framing in security ML.

What's already shipped

• `triage-data-collector.ts` produces JSONL with text + 45-feature vector + label + provenance (commit 5bc8503)
• `feature-extractor.ts` 45-feature handcrafted vector explicitly inspired by VulnBERT's hybrid architecture (the file docstring says so)
• npm-bench preserves raw findings so the data can be collected post-hoc (commit 5bc8503)
• 18 vitest tests covering the collector and feature extraction (commit 5bc8503)

What's still needed for the paper

• Grow npm-bench from 81 → ~250 packages so the labeled set is publication-scale
• First full dataset dump committed under `packages/benchmark/results/triage-dataset-v1.jsonl` (or shipped to Hugging Face)
• Baseline classifier (CodeBERT-scale or XGBoost on the 45-feature vector alone) trained and reported
• Hybrid classifier (handcrafted features + neural embeddings, VulnBERT architecture) trained and reported
• Joint pipeline: pwnkit raw → classifier → triaged finding, benchmarked against pwnkit alone on `npm-bench` (currently F1 0.444)
• arXiv preprint (cs.CR)
• Hugging Face: dataset card + model card
• Blog post on pwnkit.com / opensoar.app

Honest negative results to include

• Handcrafted features designed for web-exploit findings (SQL errors, payload reflection, stack traces) are mostly zero on npm supply-chain findings — a publishable insight on domain transferability
• `label_source: package_verdict` is coarser than per-finding labels and produces some false-FPs on safe packages with legitimate findings — quantify the noise floor

Sequencing

This issue intentionally has no deadline. The work is the moat and the research direction is the long game.

Why this issue exists

So the work is in the open and not in someone's notebook. Discoverable, citable, mergeable.

[peaktwilight]

Call with @quguanni happened (2026-04-10)

30min Calendly call. Posting the takeaways here so the trail is in
public for anyone tracking the joint paper plan.

What she said

1. Wants to make VulnBERT itself better first before plugging it
   into pwnkit. v9 is in progress; we don't have details. Not ready
   to ship a v8.5 against pwnkit's dataset yet.
2. Believes bigger LLMs are the future for the text-encoder
   half of the hybrid architecture. Open question whether a fine-tuned
   Llama-8B (or larger) beats CodeBERT-125M for embedding finding text.
   This is hers to evaluate, not ours.
3. Impact matters to her — wants results that move the field, not
   incremental improvements. Negative results count if they're
   methodologically rigorous.
4. Liked the A/B / feature-flag approach. Sees value in rigorous
   ablation. This validates pwnkit's existing PWNKIT_FEATURE_*
   infrastructure as a publishable methodology, not just a config knob.
5. Wants to know what data + artifacts we'll prep for her side.
   That's the question driving the next 30 days of work on this issue.

What we own on the pwnkit side over the next 30 days

The collaboration is sequenced: her side is busy improving VulnBERT, so
the pwnkit half owns prep — not integration.

#	Deliverable	Status	Tracking
1	Per-layer ablation on stubborn-14 (4 profiles, white-box)	✅ landed	#72
2	Per-layer telemetry (layerVerdicts per finding)	not started, prerequisite for #4	new issue (TBD)
3	Per-layer ablation on a less-stubborn limit=50 slice	not started	#72 follow-up
4	triage-dataset-v1.jsonl from existing benchmarks	half-built (collector exists, needs telemetry)	this issue
5	Dynamic routing prototype design doc (the novel idea)	not started	new issue (TBD)
6	npm-bench recall analysis: per-case breakdown of the 14 misses	not started	new issue (TBD)

The dataset is the moat — re-confirmed

Sequencing the next 30 days as a prep sprint, not an integration sprint:
the per-layer telemetry has to land before the dataset is meaningful, and
the dataset has to land before there's anything useful to ask Guanni for.
Don't ping her with status updates — wait until we have the dataset +
ablation numbers in hand and then send a single "here's what we've got,
what do you need from us next?" message.

Honest input from #72

The first per-layer ablation landed today
and it's a negative result: the moat layers regress solve rate on the
stubborn-14 slice (4 → 0 flags as more layers are added, costs 2.4x
more). That's exactly the kind of empirical finding the joint paper
should center on — which layers actually move solve rate, on which
slices, at what cost? — rather than vendor "95% accuracy" marketing
claims.

What the novel paper contribution looks like now

Not just "hybrid features + embeddings beats either alone" — VulnBERT
already published that on kernel commits. The web-domain version is
the obvious extension and would be table-stakes.

The differentiator is learned dynamic triage routing: instead of
statically toggling triage layers via env vars, train a small model
that decides per finding which layers to run. Training signal comes
from per-layer telemetry (deliverable #2 above). Nobody has published
this — every existing hybrid system (SAST-Genius, Datadog Bits AI,
Endor Labs) uses a static pipeline. The negative result from #72 is
the motivation: if some layers help on some findings and hurt on
others, a learned router beats any static threshold.

That's the contribution worth co-authoring on.