AI-Generated Code Security: Closing the Governance Gap

AI-generated code security accountability belongs to the organization producing the code because its development system controls the authorship record, review evidence, acceptance decision, and attestations that auditors can examine after an agent-authored change ships.

AI-Generated Code Security in Practice

In a hypothetical example, an agent adds a retry wrapper around a payments client, CI passes, and an engineer merges the pull request after a ninety-second review. Fourteen months later, a SOC 2 Type II auditor samples that commit and asks who wrote it, what the reviewer examined, and what the organization knew about the tool that produced it.

The commit metadata names an engineer who accepted a suggestion he no longer remembers, and the diff contains a connection string with embedded credentials. No artifact records which model and version produced the change or what context it received.

This guide is for engineering managers, security leads, compliance owners, and software auditors. Pipeline control placement and scanner or platform evaluation sit outside its scope.

Why AI Authorship Is Hard to Prove

Current frameworks do not require an AI-authorship field, even though SLSA and Sigstore already support automated builders and workload signers. That mismatch leaves organizations with mature build attestations but no standard field showing how an agent produced a particular source change.

The Accountability Gap in Agent-Authored Code

Most source-code accountability instruments still rely on human-centered records. Git records an author field that, for agent-written changes, names the person who accepted the output, and code review sign-off traditionally confirms that a peer examined another person's reasoning. The CISA attestation form, released in March 2024 under the procurement requirement OMB M-22-18 and M-23-16 established, covers secure development practices under NIST SP 800-218 without asking whether AI produced the code.

Standards bodies have made the gap explicit. NIST published NIST SP 800-218A in July 2024, and the profile does not distinguish human-written from AI-generated source code because it assumes teams evaluate all source code before use. The SLSA project released SLSA v1.2 on November 24, 2025, defining build provenance in general terms with no AI-authorship category. In the EU, the DORA risk management regulation requires static and dynamic source code review in Article 16(3) without distinguishing who or what wrote the code.

Silence does not lower the bar. With no framework carving out AI authorship, the organization carries the review and evidentiary burden regardless of author. An agent cannot explain its past reasoning, so the development record must answer an auditor's questions.

Why LLMs Ship Insecure Patterns at Scale

The AI-generated code security risks that reach production start in the training corpus, because corpus quality shapes generated-code quality before a prompt is written. The strongest controlled intervention comes from Improta et al. (ICPC 2025), who trained one model on an uncleaned corpus and another on a cleaned corpus. The first generated low-quality functions 5.85% of the time and the second 2.16%, while both preserved functional correctness. Improta et al. demonstrated a causal effect on code quality, though their experiment did not measure vulnerability inheritance directly.

Optimization compounds that quality problem. The CWEval benchmark from January 2025 measured the gap between code that works and code that works securely. Moving from functional correctness to functional-and-secure correctness reduced the pass@10 rate by roughly 30% across the five mid-tier models tested. Veracode's 2026 GenAI Code Security Report, published July 28, 2026, puts the average security pass rate at 56% across four snapshots and more than 100 models, with 11 new models tested across 80 function-completion tasks in the 2026 edition. Veracode scores the generated code with its own SAST product, and each task is built so the requested functionality can be written either securely or insecurely.

The Vulnerability Classes That Recur

Recurring agent-authored weaknesses map onto the frameworks reviewers already use, MITRE's 2025 CWE Top 25 from December 2025 and the OWASP Foundation's Top 10:2025 release candidate from November 2025. OWASP places cross-site scripting and SQL injection under A05:2025 Injection, hardcoded secrets under A07:2025 Authentication Failures, and authorization weaknesses under A01:2025 Broken Access Control.

Each row carries its own caveat. The CVSS figure comes from Schreiber and Tippe's 2025 mining study, a preprint that puts the top five CWEs within 0.19 points of each other. The Apiiro telemetry behind the access-control row used static analysis without establishing exploitability, and the same analysis found AI-assisted developers reduced syntax errors by 76%. GitGuardian's State of Secrets Sprawl 2025 identified Copilot-enabled repositories from configuration files, so its 6.4% measures leakage where the tool was active rather than leakage the tool authored.

The secrets row is asymmetric. MITRE's 2025 analysis reports that CWE-798 fell out of the Top 25, yet hardcoded credentials remain among the highest-severity findings in AI-attributed code. Population-level rankings and agent-authored code diverge, which supports tracking authorship.

What Producing Evidence Requires

None of what follows can be created retroactively, so it has to exist before an auditor samples anything.

• Named engineering and compliance owners for authorship records
• Development-record and scanning systems that hold and export evidence linked to a change
• Machine-readable export formats tied to individual changes across agent sessions, repository events, review decisions, and scan output
• An approved list of AI coding tools
• Retention periods covering the organization's audit and incident-review windows

Where any of it is missing, an auditor's questions get answered from an engineer's memory instead of from a system.

From Authorship Context to an Evidence Package

An auditable record spans the arc from the moment an agent receives context to the point at which the organization can present a reviewable package, and each stage holds something later stages cannot recreate.

Recording Authorship Context

The record should carry the model and version that produced the change, the prompt and context it received, the policy governing the agent, and the human who accepted the output. None of it can be reconstructed later, because model retirement and session expiration erase unlogged context.

• Change-level record: Record prompts, governing policy, scan output, and human acceptance against the specific change. An AI-BOM does not replace this session-level evidence.
• AI component inventory: OWASP defines an AI-BOM as a machine-readable inventory of AI components with evidence of origin, rights, integrity, and evaluation. Ecma International ratified CycloneDX v1.7 as ECMA-424 second edition on December 10, 2025. That makes it the only AI/ML-BOM-capable format a formal standards body has ratified, since ISO/IEC 5962:2021 covers SPDX 2.2.1 only. CISA and G7 partners published minimum AI SBOM elements on May 12, 2026, describing them as neither exhaustive nor mandatory.
• File provenance: C2PA v2.4 added manifest embedding for source code and an AI Disclosure Assertion, c2pa.ai-disclosure, which no jurisdiction mandates.

Specifications describe each artifact, but the layer running the agent must produce the change-level record, because that layer holds the authoring context. Cosmos, Augment Code's unified cloud agents platform, pairs Environments, which define where agents run and what they can access, with Experts, which package the judgment a task requires; Sessions retain the account of each run for later audit.

Teams evaluating agent orchestrators should make authorship-time capture a first-order requirement, because a record added at merge misses the context that existed before the commit.

Shared Responsibility Across Layers

Responsibility spans provider duties, developer accountability, organizational governance, and independent review. No published framework allows one layer to absorb the others.

Vendors carry baseline duties: the secure AI guidelines CISA and the UK NCSC published on November 26, 2023, sealed by 23 organizations, prioritize provider ownership of security outcomes. Developers carry the next layer, and the OWASP AI coding guidance assigns responsibility to whoever accepts and commits generated code, because the tool accepts none.

Organizations hold the governance layer. NIST AI 600-1, published July 2024, sets out generative-AI risk-acceptance policies and incident-response ownership organizations can adopt. The Cloud Security Alliance's AI Controls Matrix v1.1 of June 22, 2026 distributes 247 control objectives across 18 security domains and defines five accountable roles, from model provider through AI customer. Independent reviewers then test whether those layers produced usable evidence.

Binding Attestations to Systems

Supply-chain attestation already handles non-human signers. What changes is the human review, because the reviewer must attest to the recorded process around an agent-authored change.

SLSA's provenance predicate requires a buildType, externalParameters, and a builder id, and the specification separates builder and signer identities so one signer can attest for multiple builders. Existing attestation systems therefore do not assume a human signer, even though SLSA lacks an AI-authorship field.

Sigstore extends the model to workload identity. Fulcio binds ephemeral keys to OIDC identities, and those keys exist only in memory and never reach disk, so a GitHub Actions workflow can sign as itself. GitHub Artifact Attestations issue certificates valid for ten minutes, and the client discards the private key after signing.

For agent output, the reviewer attests that the recorded provenance matches policy, that the diff was examined, and that the attached checks passed. A Cosmos Session can preserve the work record and acceptance event, while the repository preserves the resulting change and review decision.

Governance Controls for Non-Human Authorship

Regulators evaluating non-human authorship look for due-diligence structure, and today's controls reach it obliquely.

• Acquisition controls: Control SC.06 of the CISA acquisition guide is the most concrete public description, covering an approved list of AI code-generation implementations, security-function risk assessments, automated identification of AI-generated code, and periodic data-leakage review. Security teams should start from a defensible shortlist of AI coding tools.
• Payment guidance: The PCI AI principles, published September 11, 2025, state that AI may perform the individual actions that create, test, and deploy software, but that the pipeline as a whole should not be fully automated. They are guidance and do not amend PCI DSS v4.0.1.
• EU transparency scope: Article 50(2) of the EU AI Act requires machine-readable marking of synthetic audio, image, video, and text from August 2, 2026. The Digital Omnibus regulation, in force July 27, 2026, extends that to December 2, 2026 for systems already on the market. The Commission's Article 50 FAQ places source code outside the obligation.
• Financial-sector testing: BaFin's DORA guidance, dated June 2024, confirms static and dynamic testing before productive use, which Article 16(8) extends to code third parties create.

Cosmos Environments give teams a defined place to scope where agents run and what they can access, while the organization's review controls remain responsible for approval.

Attaching Security Evidence to the Record

Agent output accumulates faster than review models built for human throughput can clear it. Liu et al. tracked 302,579 verified AI-authored commits across 6,299 repositories and found 22.7% of the 484,366 issues those commits introduced still surviving at HEAD. Their net-impact finding cuts sharper for a review process: AI-authored commits fix slightly more code smells than they introduce, and introduce more correctness and security issues than they fix.

Treat SAST and SCA as evidence generators that feed the authorship record. Whatever enterprise SAST tools a team already runs, the orchestration layer should attach each result to the change at authorship time, because a finding that lives only in a dashboard cannot support a later evidence package.

Deep Code Review, one of the specialized Experts in Cosmos, reads a pull request end to end, posts inline findings, and retains the review activity in the associated Session. Platform selection for this layer is a separate exercise, and AI governance tools covers those criteria.

Packaging the Evidence

A package an auditor can work from separates what was captured from where it lives and how long it survives.

• Authorship record: the model and version, prompt context, governing policy, accepting human, and authoring-time scan results
• Storage locations: the Cosmos Session history, repository records, and any external evidence store
• Retention and assurance: the retention policy, with vendor certifications such as SOC 2 Type II and ISO/IEC 42001 kept separate from code-level evidence
• Audit package: the records in a format the reviewer can inspect and correlate

The CSA's AI customer audit guidance directs auditors to request prompt templates, AI configuration scripts, Git logs, and code snippets showing review comments. Cosmos Sessions hold the first two; the repository and the scanners hold the rest. Together they let the organization name what wrote a sampled change, what it saw, what governed it, who accepted it, and what checks it passed.

Troubleshooting Accountability Gaps

Two accountability gaps recur when teams retrofit human-era accountability onto agent output. Teams close them by reconnecting the authoring event to the later attestation.

Reviewers Over-Trust Fluent Output

Perry et al. (ACM CCS 2023) found that AI-assisted participants in a 47-person lab study wrote insecure solutions more often on four of five tasks. On the SQL task, 36% produced vulnerable code compared with 7% of controls, and AI-assisted participants were more likely to believe their code was secure.

Tie review depth to change risk, affected data, and reachable privileges. Do not use production speed or fluent output as a proxy for safety.

Repository Records Lack Context and Review Evidence

Git captures the acceptor and timestamp while the model version and prompt context disappear when an unlogged session closes, and a sign-off becomes a rubber stamp when it records approval without showing what the reviewer examined. Capture provenance in the layer that runs the agent before the commit exists, then bind each attestation to the diff, scan output, recorded session, and governing policy, so the signature asserts a claim an auditor can test.

What to Do Next

The tradeoff is recording at authorship versus attempting reconstruction after the fact. Recording requires process and tooling changes now, and missing prompts and expired session context can make later reconstruction impossible. Where the gap sits in the authoring record itself, it closes in the layer that runs the agent, which is what a Cosmos Session is capturing while the work happens.

Pull the ten most recent merged agent-authored pull requests this week. For each one, identify the model, version, context received, and accepting human using only existing artifacts. Score each pull request out of four. Treat every score below four as a concrete logging, policy, retention, or orchestration gap.

Frequently Asked Questions

Related Reading

• AI Incident Management Software: 2026 Evaluation Guide
• 9 Platforms with AI Root Cause Analysis: What to Look For
• Best DAST Tools for Enterprise Security (2026)
• 7 Best AI Agent Observability Tools for Coding Teams in 2026
• 8 Best AI Coding Assistants [Updated May 2026]