Skip to content

README.md

Latest commit

 

History

History
31 lines (25 loc) · 1.55 KB

README.md

File metadata and controls

31 lines (25 loc) · 1.55 KB
Raw

Historical real-bug benchmark corpus

This directory contains source-backed before/after cases from public project history. Every case records the upstream repository, public fix/advisory, exact commits, source-backed snapshots or a labeled syntax-complete excerpt, expected finding IDs, and a deterministic state invariant.

Current corpus:

Project Language Upstream fix Category
Requests Python GHSA-j8r2-6x86-q33q security state guard
FastAPI Python #5465 resource lifecycle
Prometheus Go #1070 resource lifecycle
Kubernetes Go #29495 resource lifecycle
Gin Go #4422 resource lifecycle
Terraform Go #38585 resource lifecycle

Run the complete corpus with:

smartbench benchmark run \
  --manifest benchmarks/real/manifest.yaml \
  --output benchmark-report.json

Passing means that every historical buggy snapshot produces exactly one expected finding and every fixed snapshot produces none. Full-file fixtures are preferred; the Requests case uses a syntax-complete scoped function excerpt and labels that boundary in its manifest. The corpus demonstrates deterministic expression of these known defects; it does not yet establish general precision or recall.