Why We Don't Fully Trust Our Own Benchmark

XploitScan's public benchmark currently reports 86.3% precision from our Nov 3, 2026 hand review, updated for the Nov 4 rule fixes, which counts false positives on clean fixtures. The corpus is 649 labeled code fixtures that we keep adding to (the benchmark page has the live count), with 97.2% recall. On our own benchmark (Nov 3, 2026), Semgrep's community rulesets catch about one in eight of the vulnerabilities (12.4% recall) and Bearer a bit over one in four (27.6%).

Those are real numbers. Checked daily and rerun when rules or fixtures change (at least weekly); updates go live with the next deploy. The per-rule results and the competitors' tool versions are published on the benchmark page. We're proud of them.

And you should be a little suspicious of them. We are.

The problem with acing your own exam

A detection benchmark is only as honest as its corpus. Ours is hand-curated: when we write a rule, we write fixtures — a vulnerable example the rule should catch, and a safe counter-example it must not flag. That's genuinely good engineering. Every false-positive fix ships with a clean fixture that locks the fix in. 86.3% precision from our Nov 3, 2026 hand review, updated for the Nov 4 rule fixes, which counts false positives on clean fixtures.

But it has an obvious failure mode: the people who wrote the rules wrote the test. A fixture written by the rule author tends to look like the pattern the rule already matches. Score 97.2% on that and you've proven the rule catches the thing it was built to catch — not that it catches the same vulnerability when a stranger writes it three ways you didn't think of.

This is the single most fair criticism of any security scanner's self-reported numbers, and we'd rather raise it ourselves than have a skeptical buyer raise it for us.

Two things that keep the number honest

It's a floor, not a trophy. The scores on this 649-fixture corpus are not a claim that we catch everything in production code. They are the line below which detection quality is not allowed to regress without us noticing. On pull requests that change rules or fixtures, a per-rule F1 drop of more than two points fails CI. The corpus keeps growing, so the test gets harder over time, not easier. You can watch the trend on the benchmark page.

The published precision is a hand review. XploitScan has an optional AI false-positive reviewer, and the benchmark scores the scanner with that reviewer off. 86.3% precision from our Nov 3, 2026 hand review, updated for the Nov 4 rule fixes, which counts false positives on clean fixtures. Recall is computed on every benchmark run.

The real fix: code we've never seen

The only way to answer “but does it work on code your rule authors didn't write?” is to test against exactly that. So we're building a held-out corpus from public, intentionally-vulnerable projects that no one on our side authored: OWASP NodeGoat, Juice Shop, the Damn Vulnerable NodeJS Application, and a prototype-pollution example.

These are real files with real, documented vulnerabilities — SQL injection through a raw Sequelize query, command injection through exec, XXE through libxmljs with external entities enabled, insecure deserialization, prototype pollution, hardcoded crypto keys — written by other people, years before our rules existed. We strip the hint comments these teaching projects include (so the scanner can't cheat off a // vulnerable marker) and score against the bare code.

We expect our recall to be lower on that set than on our own fixtures. That's the point. A held-out number that's a little worse and a lot more believable beats a self-graded number that looks perfect and easy to dismiss. When it lands, it'll be on the same public benchmark page, scored the same way, with the sources cited.

Update: the held-out set launched on June 10 at 6 of 10. Over the next day we changed rules to catch the misses we had studied, and we kept doing that as cases were added, so today's held-out figure includes cases we tuned against and is no longer a fully blind number.

Why we're telling you this

Most security tools show you a precision number and dare you to disprove it. We'd rather publish the per-rule results, the competitors' tool versions, and the most honest critique of our own methodology, and let you check. The whole pitch of a scanner is that you can trust what it tells you. That trust has to start with how we talk about our own results.

Want to see where the numbers stand right now? The benchmark page is live. Or just point the scanner at your own code — npx xploitscan@latest scan ., no signup, runs locally.