ai-cybersecurity/vendor
AWS benchmark shows AI code review is a coin flip
AWS released Deception Benchmark, a dataset of 14,822 code samples across 16 languages and more than 70 CWE categories, designed to test whether AI models can tell real vulnerabilities from code that looks risky but is actually safe. Twelve models from five providers were evaluated. Under standard prompting, precision landed in the mid-50s, no better than guessing. The benchmark deliberately includes only samples that successfully deceived at least one frontier model during construction, so the difficulty is calibrated to the state of the art.