What Data Shows on AI-Generated Code Security

Verified research from 2023 to 2025 shows AI generated code ships features fast but often carries security flaws without review. Formal verification of 331000 C programs reports at least 62.07 percent vulnerable as of 2024, Python benchmarks focused on security report over 75 percent insecure as of 2024, while real world GitHub analysis finds 27.3 percent of 733 AI snippets with weaknesses as of 2023 to 2024. None is a universal rate for all AI generated code.

Update, October 2026: an earlier version of this post framed the controlled and real world rates as upper and lower bounds of the same quantity and described LyraShield billing as roadmap work. These studies measure different populations, languages, models and detection methods, so their rates are not bounds of one estimate and do not establish the effect of review. None of them measures LyraShield detection accuracy, prevalence across all AI built apps or the security of current model versions.

Controlled benchmark results

Controlled studies use neutral zero shot prompts and formal methods to label flaws. They measure what models do without project context or reviewers.

FormAI at PROMISE 2023 published 112000 C programs generated by GPT 3.5 turbo with dynamic prompting. Using ESBMC formal checker, authors classify at least 51.24 percent vulnerable as of 2023, see FormAI at PROMISE 2023. ESBMC provides counterexamples that reduce false positives.

FormAI v2 expands to nine models including GPT 4o mini, Gemini Pro 1.0, Falcon 180B and Code Llama 13B with 331000 compilable C programs as of 2024. Each program is labeled through unbounded verification. The study reports at least 62.07 percent vulnerable with similar patterns and 42 CWE types including six in MITRE Top 25 2023, see FormAI v2. This figure is a lower bound under that method, not a worst case estimate.

Security focused benchmarks raise rates when tasks target risky code. A study of four models across 67 CWE types on SecurityEval reports 76.2 percent insecure on average and over 75 percent vulnerable as of 2024, see the SecurityEval study. Georgetown CSET tests five LLMs with prompts likely to elicit Top 25 patterns and reports about 48 percent compilable but buggy and about 30 percent verified secure as of 2023, see Georgetown CSET. It also summarizes earlier results such as Pearce et al 2021 reporting about 40 percent of 1689 Copilot programs vulnerable to the MITRE 2021 Top 25.

For verification and access control for vibe coding teams, see our pillar guide at /blog/vibe-coding-security-guide which frames review, testing and output handling before release.

Observational repository samples

Real world studies collect code that already passed some review and tooling; the rates in these samples are lower, but the samples also differ in population, languages and detection method.

A study of 733 AI generated snippets from GitHub projects using Copilot and similar tools reports 27.3 percent contain at least one security weakness as of 2023 to 2024, see the 733-snippet GitHub study. Python shows 29.5 percent with weaknesses and JavaScript shows 24.2 percent in that sample. Those 200 vulnerable snippets contain 628 weaknesses across 43 CWE types, average 3 per vulnerable snippet and up to 22 in a 1595 line file. Eight map to the 2023 Top 25 including CWE 330 insufficiently random values, CWE 94 code injection, CWE 79 cross site scripting and CWE 78 OS command injection.

A 2025 analysis of 7703 files attributed to AI tools reports 4241 CWE instances across 77 types using CodeQL, see the 7703-file CodeQL analysis. It reports 87.9 percent with no CWE mapped flaw, leaving a remainder. As of 2025 language split is Python 16.18 to 18.50 percent, JavaScript 8.66 to 8.99 percent, TypeScript 2.50 to 7.14 percent in the same 7703-file CodeQL analysis. Copilot shows better density for Python at 1739 lines per CWE while ChatGPT shows better for JavaScript and about 39 percent are AI generated docs.

These two views measure different settings, not an upper and lower bound of the same quantity. Zero shot isolated tasks and reviewed code with standards differ in population, languages, models and detection method, so the gap between their rates cannot be read as the effect of review. What holds across both views is the recurrence of CWEs tied to randomness, injection, XSS and resource cleanup.

For a release focused companion, see AI generated code production safety which covers auth, input validation and secrets review.

Study as of date	Scope and method	Key rate reported	Source
FormAI v2 as of 2024	331000 C programs 9 LLMs ESBMC unbounded	at least 62.07 percent vulnerable	https://dl.acm.org/doi/10.1007/s10664-024-10590-1
SecurityEval focus as of 2024	4 LLMs 67 CWEs Python tasks	76.2 percent insecure avg over 75 percent vulnerable	https://arxiv.org/html/2408.10495v1
CSET evaluation as of 2023	5 LLMs Top 25 prone prompts ESBMC	about 48 percent compilable but buggy about 30 percent verified secure	https://cset.georgetown.edu/wp-content/uploads/CSET-Cybersecurity-Risks-of-AI-Generated-Code.pdf
GitHub 733 snippets as of 2023 to 2024	Copilot similar in real projects CodeQL	27.3 percent with weaknesses 29.5 percent Python 24.2 percent JavaScript 628 flaws 43 CWEs	https://arxiv.org/html/2310.02059v3

Language effects and repair signals

A 2025 multi language study evaluates 200 tasks across Python, Java, C++ and C with five LLM families. It checks syntax, correctness with 4000 tests per program and security with SonarQube and CodeQL plus manual review as of 2025, see the 200-task multi language study. Python and Java achieve higher correctness and fewer findings than C and C++ where memory safety, secrets and crypto misuse are more common. Many models fail to use modern features such as Java 17 APIs and outdated methods remain common in C++.

A study of 108 Java questions compares StackOverflow and ChatGPT. Using CodeQL it reports 248 flaws in ChatGPT versus 302 in StackOverflow, 20 percent fewer with 25 percent overlap as of 2024, see the 108-question StackOverflow and ChatGPT study. ChatGPT shows 19 CWE types versus 22 in StackOverflow in that sample. Both human and AI sources propagate insecure patterns without verification.

Repair data shows tooling plus review beats single shot fixes. GPT 3.5 and GPT 4 achieve 33.2 percent to 59.6 percent repairing insecure code from four LLMs, but fix their own code less often, indicating blind spots as of 2024, see the SecurityEval repair analysis. With iterative feedback assisted by semantic engines, success rises to 65.9 percent to 85.5 percent as of 2024 in the same SecurityEval repair analysis. Detection false positives are higher for LLMs. GPT 3.5 reports 4.3 percent and GPT 4 reports 3.1 percent versus CodeQL and Bandit below 1 percent as of 2024 per the SecurityEval false-positive figures. In the GitHub 733 study, basic fix command resolves 19.3 percent of weaknesses while an enhanced prompt with the warning resolves 55.5 percent as of 2023 to 2024 in the 733-snippet fix-rate findings.

What this means for release assurance

Treat AI output as untrusted input to the rest of your system. OWASP Top 10 for LLM Applications 2025 defines LLM05 Improper Output Handling as insufficient validation of model outputs before they reach browsers, shells, SQL or other agents, see the OWASP Top 10 for LLM Applications project. The companion Gen AI page lists XSS, SQL injection and SSRF as downstream effects when handling is missing as of April 2024, see OWASP Gen AI LLM05. GitHub agentic principles as of November 2025 recommend visible context, firewalling, limiting sensitive access, preventing irreversible changes and clear attribution, see GitHub’s agentic security principles.

Isolate risk by language and path. Keep auth, crypto and input handling in memory safe languages with platform secrets and crypto APIs where possible. Where C or C++ is required, prioritize memory safety review because FormAI and the multi language study report highest density there.

Add deterministic checks at release, not only in the IDE. Linters, CodeQL or Semgrep rules, secret scanning and tests for auth boundaries catch repeat offenders seen in the datasets. Keep fixes as proposals requiring review, not silent automatic rewrites, so humans stay in the loop on security changes.

LyraShield AI fits this gap as release assurance for AI built apps. LyraShield AI is at https://lyrashieldai.com with open registration in open beta. As of October 2026, the scan engine, approval gated fix proposals and paid plans are live; deeper fix PR automation remains on the near term roadmap. We publish this to set honest expectations while building in public.

Use the AI app security checklist for a concrete pre release pass that maps to CWEs reported in these studies and pair it with /blog/vibe-coding-security-guide and /blog/ai-generated-code-production-safety. When teams measure real rates, use language aware controls and validate outputs with review and tooling, they keep AI speed while lowering the security cost documented from 2023 to 2025.