PreviewYou're previewing CTO AI Forge. Sign in to use it.Sign in
← All lessons
L4 · Agentic engineering: agents in CI/CD, guardrails, quality and security9 min read

Securing AI-written code

AI-generated code inherits the patterns in its training data, including insecure ones, and it is produced faster than humans can carefully read it.

What the evidence shows

  • Insecure choices are common. Veracode's 2025 GenAI Code Security Report tested 80 curated coding tasks across more than 100 LLMs. When a task could be done securely or insecurely, models chose the insecure option 45% of the time. Java had a security failure rate over 70%, and models failed to defend against cross-site scripting in 86% of relevant cases. Larger models did not do significantly better.
  • Hallucinated packages. Researchers analysing 576,000 code samples (USENIX Security 2025) found that the average share of hallucinated package names was at least 5.2% for commercial models and 21.7% for open-source models, with 205,474 unique made-up names. Attackers can register those names and wait for someone to install them ("slopsquatting").
  • Secrets and data. Assistants can echo secrets from context into code or logs if not controlled.

Controls that work

  1. Same gates for all code: SAST, dependency (SCA) and secret scanning as required checks on every PR — human or agent.
  2. Package controls: an internal proxy or registry with allow-lists; block unknown or brand-new packages by default; alert on first-time dependencies.
  3. Secure prompting defaults: organization-level instructions that require parameterized queries, output encoding, approved crypto and auth libraries.
  4. Risk-based review: security-sensitive paths (auth, payments, crypto, input handling, infrastructure-as-code) require review by a qualified engineer, whatever wrote the code.
  5. Secure SDLC alignment: NIST's SP 800-218A adds AI-specific practices to the Secure Software Development Framework.
  6. Measure: vulnerabilities found per 1,000 changed lines, split by AI-authored vs human where your tools allow.

What this means for the CTO

Assume AI-written code is at least as risky as code from a fast new hire who learned from the internet. Make security gates non-optional for every PR, add package controls against hallucinated dependencies, and require qualified review on security-sensitive paths.

Work with your CISO (see the CSO hub) to track whether vulnerability rates change as the AI-authored share rises; that trend is the evidence for — or against — your current controls.

Report that trend to the board alongside productivity: speed without security evidence is an incomplete story.

Real example

Veracode's 2025 GenAI Code Security Report (July 30, 2025) found that across 80 coding tasks and more than 100 LLMs, models chose the insecure option 45% of the time, with Java failing security checks over 70% of the time and cross-site scripting defenses failing in 86% of cases. A USENIX Security 2025 paper found at least 5.2% (commercial) and 21.7% (open-source) of suggested packages were hallucinated, across 576,000 samples. Source: Veracode — AI-generated code poses major security risks in nearly half of all development tasks (Jul 30, 2025) ↗ · Spracklen et al. — We Have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating LLMs (USENIX Security 2025; arXiv 2406.10279) ↗

Use it

Quick check · 3 questions

  1. 1.What is 'slopsquatting'?

  2. 2.According to Veracode, did larger models produce significantly more secure code?

  3. 3.Which control best addresses hallucinated packages?

0 of 3 answered

Sources

Last reviewed 2026-10-01.