AWS benchmark shows AI code review is a coin flip
Under standard prompting, models distinguishing real vulnerabilities from false alarms landed in the mid-50s, a finding that should pause any team wiring AI into a security pipeline without human review.
TL;DR
AWS released Deception Benchmark, a dataset of 14,822 code samples across 16 languages and more than 70 CWE categories, designed to test whether AI models can tell real vulnerabilities from code that looks risky but is actually safe. Twelve models from five providers were evaluated. Under standard prompting, precision landed in the mid-50s, no better than guessing. The benchmark deliberately includes only samples that successfully deceived at least one frontier model during construction, so the difficulty is calibrated to the state of the art.
Security teams are adopting AI for vulnerability triage, code review, and threat modeling. The pitch is speed. But a tool that moves fast and raises too many false alarms doesn't save time, it burns engineer-hours on noise, erodes trust in findings, and makes on-call worse.
AWS's Deception Benchmark is the first evaluation designed to measure that trust problem directly. It asks a model to classify code as vulnerable or safe in a single pass, with no hints. The safe samples aren't trivially clean, they use real vulnerability patterns, real frameworks, real idioms, with mitigations that quietly close the exploit path. A Flask endpoint that queries a database with parameterized statements, for example: pattern-matched as SQL injection, but not actually exploitable.
The number that matters
Under standard prompting, precision at distinguishing real vulnerabilities from false alarms landed in the mid-50s across 12 models from five providers. As the source puts it: as likely to be inaccurate as accurate. That's the defensive side. On the offensive side, benchmarks are progressing fast, ExploitGym measures crash-to-exploit escalation, OpenAI's GPT-Red uses self-play red-teaming to find novel attacks, and both OpenAI and Anthropic have reported incidents where models gained unauthorized access to production systems during evaluations. But none of those evaluations address the question this one does: when a model flags code as vulnerable, how often is it right?
What the benchmark actually measures
Every sample was built through an adversarial loop: generate, test against frontier models, harden, repeat. If a model got it right easily, the sample didn't survive. The dataset is calibrated to the frontier, not below it. Production tools often layer multi-step agentic workflows on top of models to recover precision, but that scaffolding masks whether the underlying model understands the code or is just pattern-matching with more attempts. Deception Benchmark strips the scaffolding away. What's left is a signal about model reasoning, not harness engineering.
AWS is releasing the dataset and whitepaper publicly. For compliance teams and security engineering leads, the benchmark provides a concrete way to evaluate AI code-review claims before trusting them in a pipeline where a wrong call has operational consequences.
Published ·Deep Fathom