vendor
AWS benchmark shows AI code review is a coin flip
Under standard prompting, models distinguishing real vulnerabilities from false alarms landed in the mid-50s, a finding that should pause any team wiring AI into a security pipeline without human review.
AWS released Deception Benchmark, a dataset of 14,822 code samples across 16 languages and more than 70 CWE categories, designed to test whether AI models can tell real vulnerabilities from code that looks risky but is actually safe. Twelve models from five providers were evaluated. Under standard prompting, precision landed in the mid-50s, no better than guessing. The benchmark deliberately includes only samples that successfully deceived at least one frontier model during construction, so the difficulty is calibrated to the state of the art.