Open SAST benchmark · Apache 2.0 · no signup
BenchProctor is an open benchmark corpus for static analysis. Every test case is labeled vulnerable or safe across 11 languages, with no category or label hint in its name or identifiers. Score compatible SARIF 2.1.0 output with the bundled Python scorer.
Release 2026.07.22 is live now: 2,938,418 balanced cases across all 11 languages. Read the release update →
$ python score_sarif.py results.sarif expectedresults-2026.07.22.csv # 2,938,418 cases | 11 languages | 242 categories CATEGORY-AVERAGED SCORING (macro-averaged) FLAT AGGREGATE SCORING SUMMARY reports TPR, FPR, and Youden's J Actual scorer section headings. Your metrics depend on your scanner output.
Why another benchmark
PROBLEM 01
Most suites ship a fixed set of human-written cases. The tools, and the data they were trained on, overfit to them, and a high score stops meaning real-world accuracy.
→ Versioned releases use fresh variants while preserving the published scoring contract.
PROBLEM 02
When a path, identifier, or comment says sqli_true_positive_01, you are testing string matching, not analysis. A scanner can score high for entirely the wrong reason.
→ Zero comments, zero CWE tags, file names that reveal nothing. The CSV is the only ground truth.
PROBLEM 03
Real findings cross files, services, and languages, and sit next to sanitizers that almost work. Single-file, Java-only suites never exercise any of that.
→ Eleven languages and twenty-one framework targets ship standalone now. Cross-file, polyglot, and evasion shapes remain on the public roadmap.
What a test case is
Each category is one vulnerability class expressed as a taint flow. Untrusted input enters at a source, travels to a sink, and a defense either holds or it doesn't. A vulnerable case is missing an effective sanitizer; its safe twin has one. Every case is graded by difficulty, and each category comes out balanced 50/50 vulnerable to safe.
Where untrusted input enters: query params, request bodies, headers, env vars, files, message payloads.
How the data travels to the sink: concatenation, decoding, parsing, indirection, collection passthrough.
What would neutralize it: parameterization, output encoding, validation, allow-lists, strong crypto.
The dangerous call: query execution, command exec, HTML render, file open, redirect, deserialization.
# vulnerable case source http_request_body
propagator json_parse, field_access
sink sql_query
label vulnerable # safe twin, same flow, one defense that holds source http_request_body
propagator json_parse, field_access
sanitizer parameterized_query
sink sql_query
label safe The defense in a safe case always fits the sink it protects. A scanner never gets a free win from a nonsense pairing like an HTML encoder guarding a SQL query, so passing means the tool actually reasoned about the flow.
Seeded rotation
A rotated release can change the actual test code, so a tool cannot rely on recognizing prior files. The published scoring contract keeps the same category identity, balance, and coverage, making results comparable across releases.
# fresh variants, identical contract
corpus-2026.07.22/ 242 categories · 50/50 · difficulty held
corpus-next-release/ 242 categories · 50/50 · difficulty held # results stay comparable across the fixed scoring contract # prior file recognition does not transfer to the rotated corpus [ok] new code, identical contract
Anti-leakage by construction
A benchmark only measures analysis if the file can't be classified by its name or its comments. Every file in the corpus holds to these properties, so nothing is left to authoring discipline.
Emitted code carries none. There is no natural-language hint for a model to latch onto.
Not in the code, not in identifiers, not in filenames. Nothing in a test file names its own vulnerability class.
The name and position of a file tell you nothing about its category or its label. There is no ordering to read a pattern off of.
The answer key lives in one place the scanner never reads: a separate expectedresults CSV that only the scorer ever opens.
Current scope and roadmap
Release 2026.07.22 contains standalone cases across 11 languages. Broken-sanitizer variants ship now; the other shapes below remain on the public roadmap.
A planned shape that threads a weakness across modules and functions. It is not part of the current standalone release.
A planned shape for taint that crosses language and process boundaries. It is not part of the current standalone release.
A sanitizer is present but bypassed: a flawed regex, wrong-context escaping, an insufficient length limit. Scanners that trust the mere presence of a sanitizer mislabel these as safe.
Adversarial and evasion-oriented cases are in active development. They are not yet a public release claim.
Polyglot coverage
In each language, the code reads the way a developer in that ecosystem would actually write it: framework DTOs, Pydantic models, form classes, ORM calls, not toy snippets. Adding a language changes nothing about the categories, so coverage stays uniform across the matrix.
Python
Flask · Django · FastAPI
Go
net/http · Gin
Java
Spring · Jakarta EE
JavaScript
Express · Koa
TypeScript
NestJS · Express
PHP
Laravel · Symfony
Ruby
Rails · Sinatra
Bash
standalone
Rust
Actix · Axum
C
standalone
C++
cpp-httplib · standalone
OWASP Top 10 2025
213 of the 249 CWEs mapped by the OWASP Top 10 2025 are covered as SAST code-pattern test cases (85.5%). The remainder is config-level, supply-chain, or runtime-only, none of it expressible as a static code pattern (A03 is composition analysis, a separate discipline from SAST).
Scoring
Run your scanner, hand the SARIF and the answer-key CSV to a one-file scorer, read three numbers. No service, no account, no telemetry. The math is fully in your hands.
The whole formula
detected ignored
vulnerable TP FN
safe FP TN
TPR = TP / (TP + FN) detection rate
FPR = FP / (FP + TN) false-alarm rate
J = TPR − FPR Youden's JWhat the score means
Every category is weighted equally, so a handful of huge categories can't carry the score. A flat aggregate is reported alongside for comparison.
Compatible SARIF 2.1.0 output can be scored. The scorer is a single standard-library Python file with no dependencies. Matching still depends on the locations and CWE data a tool provides.
Ground truth you can trust
A benchmark is only as useful as its labels and its scoring rules. Each public bundle includes test code, an answer-key CSV, a scorer, and a manifest with integrity data. BenchProctor states that releases pass its gate suite before publication; the per-file proof metadata is not public.
# public bundle contents testcode/ the programs you scan, blind expectedresults.csv test, category, label, CWE score_sarif.py the public, dependency-free scorer benchproctor-manifest.json counts and checksums # the label for each program safe or vulnerable [ok] public labels and integrity data included
Check the public artifact
Each public label includes its category and CWE in the answer key. If you think a label is wrong, inspect the released case and answer key, then raise the specific discrepancy.
The ecosystem
Five focused tools for shipping with AI agents. Each stands on its own; together they cover code, action, orchestration, memory, and proof.
TheAuditor
Don't let agents guess about your code.
theauditortool.com →Warden
Keep AI coding work visible across supported providers.
wardenclient.com →Arbiter
The command center for your AI operation.
arbitermcp.com →Curator
Don't let agents guess about you.
curatormcp.com →BenchProctor
A public SAST benchmark built to resist label leakage.
you are hereGet started
Three commands stand between you and a reproducible score for your scanner. Releases are versioned and published periodically; the scorer is a single standard-library Python file.
# 1. run your scanner, export SARIF 2.1.0 $ your-tool scan ./corpus --format sarif -o results.sarif
# 2. score against the answer key $ python score_sarif.py results.sarif expectedresults-2026.07.22.csv
# 3. read TPR, FPR, and your Youden J → category-averaged + flat aggregate, per-category breakdown