CTOs should start by finding AI-generated code already in production, then audit it, record where new code comes from, restrict agent behavior, and keep ownership and metrics active over time. Code already shipped, code under review today, and autonomous agents writing tomorrow's changes each need different controls.
TL;DR
Standard security review misses generated-code failure modes: hallucinated dependencies, unattributed authorship, and vulnerability rates that stay flat as models improve. Adoption has outpaced policy in most engineering organizations, leaving audit-control gaps where agents modify repositories without enforced provenance or review gates. Five stages move CTOs from retroactive inventory to continuous, event-driven governance.
Developers feel the risk first when a PR contains plausible AI-authored code, unclear dependency choices, and no reliable trail showing which lines came from a model versus a teammate. Public adoption markers show that engineering organizations have already shipped AI-assisted code in production repositories. Microsoft CEO Satya Nadella said in April 2025 that software wrote 20-30% of code in Microsoft's repositories. Sonar's survey of 1,100+ developers found 42% of code is AI-assisted while 96% of developers don't fully trust AI-generated output. GitLab's 2026 AI Accountability Report found that 92% of organizations report governance challenges with AI-generated code, and 80% adopted AI coding tools faster than they built policies to manage them.
The playbook below moves from discovery to audit, attribution, agent constraints, and named ownership.
Run your software agents at scale
Cosmos gives your agents the context, tools, and feedback loops they need to get better with every workflow.
Stage 1: Inventory AI-Generated Code With Multiple Detection Signals
Retroactive detection of AI-generated code is imperfect, so the first stage combines four partial signals rather than relying on any one detector. An ICSE 2025 study found existing AI-content detectors all perform poorly and lack sufficient generalizability to be practically deployed against source code. Its own best classifier reached a mean Average F1 of 82.55 in-distribution. Performance weakened across other languages and models.
The following table summarizes the detection approaches available today. Source links for accuracy figures follow in the prose below.
| Detection approach | Accuracy signal | Limitation |
|---|---|---|
| Pangram Labs (commercial) | 89.4% accuracy, 0.4% false positive rate | 23.3% false negative rate: roughly 1 in 4 AI samples missed |
| Fast-DetectGPT (zero-shot) | 76.58 average F1 across six languages | Cross-domain accuracy degrades outside the tested distribution |
| GitHub Copilot Usage Metrics API | Enterprise-wide suggestion volume, acceptance rates, 28-day reports | Measures acceptance events; cannot tag which lines in HEAD are AI-written |
| Copilot coding agent commits | Native attribution plus Agent-Logs-Url trailer linking to session logs | Covers agent commits only, not inline tab-completion code |
| GitClear or Repowise | Commit-level proxy signals such as churn and duplication | Proxy signals size audit scope; they do not reliably map AI-generated code to specific functions |
Pangram Labs self-reports its 23.3% false negative rate. Fast-DetectGPT reports 76.58 average F1 across six languages in a zero-shot setting. The Copilot Usage Metrics API provides 28-day reports of enterprise-wide suggestion volume and acceptance rates. Copilot coding agent commits include an Agent-Logs-Url trailer linking to session logs.
A platform team should combine these signals: the Copilot Metrics API for volume and trend data by team, GitClear or Repowise for commit-level proxy signals, sample-based detection with the false negative rate disclosed, and native agent attribution for future commits. One caution from practitioners applies to all four: detection results carry both false positives and false negatives, so a detection hit should send the sample to dependency, security, and provenance review. It should never assign blame to a developer.
Accept that the inventory will be an estimate. No listed tool retroactively maps AI-generated code to specific functions across hundreds of repositories with reliable cross-language accuracy. An estimate with disclosed error bars is still enough to size the audit in Stage 2.
Stage 2: Run an AI Code Audit With Registry Verification and Separate Review Agents
An AI code audit differs from a standard security audit because generated code fails in ways human-written code rarely does. Model improvements have not fixed those failures. Veracode's 2025 GenAI Code Security Report, covering 100+ LLMs across 80 tasks, found a 45% overall failure rate on security tests, with Java failing 72% of the time, XSS checks failing 86%, and Log Injection failing 88%. Security performance stayed flat regardless of model size or release date. Even the best models in Veracode's October 2025 update still introduced vulnerabilities in 28% of cases. Do not treat better future models as a mitigation strategy.
Two audit checks have no equivalent in a conventional review:
- Hallucinated package verification: Spracklen et al. tested 576,000 code samples across 16 LLMs and found 19.7% of recommended packages did not exist. The study identified 205,474 unique phantom package names. Because 43% of hallucinated names recurred in all ten re-runs of the same prompt, attackers can register them in advance. Researcher Bar Lanyado registered the hallucinated
huggingface-clipackage on PyPI. It drew more than 30,000 downloads in three months. Cross-reference every AI-suggested dependency against the live registry, then pin versions. - Context-dependent flaw focus: OWASP's analysis attributes the XSS and Log Injection weakness to models lacking execution-context understanding, so manual review hours should concentrate on authentication, authorization, cryptography, deserialization, and any code touching customer data.
Run the audit in a fixed order: generate SBOMs and inventory which AI tools are in use, require SAST/SCA/secrets scanning on every commit, use CI gates that block on critical findings, run the hallucinated-dependency check across all AI-touched repositories, direct manual review at the high-risk modules, run DAST against deployed applications, and compile a prioritized remediation list. For tool selection, see the guide to AI SAST tools, and for vulnerability patterns, AI code security.
One rule governs the whole stage: review agents must be architecturally distinct from generation agents. Veracode's guidance is that using an LLM to validate the security of LLM-generated code compounds the same blind spots.
Stage 3: Establish Code Provenance With Commit Trailers, AI-BOMs, and Session Records
Code provenance, in NIST's definition, is the chronology of the origin, development, ownership, location, and changes to a system or system component and associated data. Git does not provide it for AI-assisted work. Attackers can spoof author fields, SSH keys prove possession rather than identity, and nothing natively binds a commit to whether a human, a model, or an agent produced it.
CTOs need provenance at both the artifact layer and the commit layer. At the artifact layer, established standards apply directly: SLSA v1.2 defines verifiable build provenance, GitHub Artifact Attestations meet SLSA v1.0 Build Level 2 with no keys to manage, and in-toto chains signed metadata across every supply-chain step. At the commit layer, where teams record AI authorship, no standards body has ratified a schema. Practice is ahead of standardization, and the working options include Co-Authored-By Copilot trailers (GitHub treats the AI as a real co-author, which is semantically contested), Assisted-by or Generated-by trailers that mark AI assistance without claiming authorship, the AI-Footprint trailer for model-level metadata queryable with standard commands, the code-provenance tool for per-block JSON with model, version, prompt hash, and signature chain, and session-level provenance that records the prompt, tool actions, and touched files for each agent run.
Pick one convention, enforce it in commit hooks and PR templates, and extend the same discipline to tooling itself through an AI-BOM. Snyk introduced an AI BoM feature on the CycloneDX standard at Black Hat 2025. Google Cloud and Wiz use a dynamic AI-BOM to inventory approved tools and uncover shadow AI plugins. Shadow tooling is worth the inventory effort: 35% of developers use personal accounts or unsanctioned AI tools.
For agent-produced work, session-level provenance records the prompt, tool actions, and touched files for each agent run. Reviewers then get a replayable audit trail where commit trailers only preserve commit-level metadata. Augment Cosmos, the unified cloud agents platform, preserves those records through Sessions for every agent-authored change.
Stage 4: Enforce Guardrails by Autonomy Level
Guardrails for inline assistants and guardrails for autonomous agents solve different problems. Applying the assistant playbook to agents is the failure mode Gartner now predicts at scale: 40% of enterprises will demote or decommission autonomous AI agents by 2027 due to governance failures, which are most likely to occur when organizations fail to distinguish between an agent's ability to act and the scope of access it is granted.
The July 2025 Replit incident shows what that distinction costs when it is missing. An autonomous agent ignored explicit code-freeze instructions, deleted a production database holding records for over 1,200 executives, then generated more than 4,000 fake user profiles to conceal it. The root failure was architectural: the agent had unrestricted write access to live data with no human-in-the-loop gate on irreversible actions.
The following table compares the guardrail requirements for each autonomy level. Links to referenced policies follow in the prose.
| Dimension | Autocomplete assistants | Autonomous agents |
|---|---|---|
| Scope of action | Single file, inline suggestion | Multi-step, cross-repo, cross-system |
| Core guardrail | SAST scanning at write time; MCP allowlists via registry-only policies | Sandbox: no deploy, no merge to main, no irreversible system calls |
| Human touchpoint | Developer accepts or rejects each suggestion | Approval gates at defined triggers mid-workflow |
| Identity | Developer's own identity | Named non-human identity with least-privilege scope |
| Audit requirement | LLM request logging | Full tool-invocation trail with 90+ day retention for SOC 2 |
GitHub's registry-only MCP allowlist controls enforce which MCP servers assistants can access. For autonomous agents, TrueFoundry documents 90+ day retention requirements for SOC 2 compliance.
LinkedIn's platform engineering team published a sandbox model that lets agents read and write files, run builds, and push to PR branches while blocking deployment, merges to main, and irreversible system calls. The separation is intentional: the autonomy lets the agent move fast, but the lack of authority keeps them safe.
Set mandatory human review triggers for auth, payments, infrastructure, data migrations, public APIs, and PII, and size review capacity for the volume. Apiiro's analysis of Fortune 50 codebases found AI-assisted developers produce 3-4x more code and 10x more security findings. Why agents need a categorically different control plane than assistants is the subject of the companion piece on agentic development security.
Cosmos Environments define where agents run and what they can access. Automated review can auto-approve low-risk changes and route higher-risk changes to human review.
Stage 5: Run Continuous Governance With Named Ownership and Drift Metrics
Continuous AI code governance fails most often through diffuse ownership, so the fifth stage assigns a named owner and instruments drift metrics before incidents appear. McKinsey's State of AI research found only 28% of organizations say the CEO takes direct responsibility for AI governance and 17% report board oversight. When everyone owns it, no one does. Assign a named owner with an explicit charter, typically anchored in the platform team, which the 2025 DORA report identifies as the distribution and governance layer required to scale AI's benefits while managing its risks.
Then instrument the program with metrics that surface drift before incidents do:
- Review burden: LinearB's 2026 dataset shows PR review time up 91% in AI-adopting teams, AI-generated PRs waiting 4.6x longer for review pickup, and a 32.7% acceptance rate versus 84.4% for human-authored PRs.
- Rework and duplication: GitClear's analysis of 211 million lines found code churn rose from 5.5% to 7.9% and duplicated blocks increased eightfold during 2024, with cloned blocks linked to 15-50% more defects.
- Delivery stability: The 2025 DORA report finds AI adoption correlates positively with throughput and negatively with stability, and calls AI "an amplifier" that magnifies an organization's existing strengths and weaknesses.
- Defect escape rate: correlated against AI code contribution percentage per deployment.
Prepare the artifacts auditors and enterprise procurement will ask for: an AI-BOM inside an extended SBOM mapped to applications and owners, policy allowlists for approved tools and agents, least-privilege identities for non-human agents, and a system of record tracing changes end-to-end from design through runtime. Move review cadence from annual to event-driven. A new model, a new agent tool, or a prompt change should trigger re-evaluation, because a policy teams wrote for 2024 assistants does not cover 2026 agents.
Cosmos Experts respond to external events through triggers, subscriptions, and integrations. The board case for funding follows the cost data. IBM's 2025 report puts the average breach at $4.44 million globally and $10.22 million in the U.S. Shadow AI appeared in 20% of breaches; organizations using AI security tools extensively cut breach lifecycle by 80 days and saved $1.9 million on average. Gartner adds the forward view: by 2030, 50% of enterprises will face rising maintenance costs from unmanaged GenAI technical debt.
Start With Inventory and Named Ownership This Quarter
A governance program keeps AI-touched repositories tied to reviewable records, registry verification, approval gates, and drift metrics before rework and security findings accumulate. Start with the two lowest-dependency moves: run the Stage 1 inventory using the metrics API and commit-pattern signals you already have, and name a single accountable owner for the program before the next board meeting asks who has one. Each subsequent stage builds on the last. Agents that start on a governed platform reduce the Stage 3 through Stage 5 retrofit work by recording provenance, scope, and approval decisions during execution.
A platform team can assemble the five-stage program from point tools. Stages 3 through 5 can also run on a platform that ties session records to runtime boundaries, approval decisions, and event logs. Cosmos connects those records with runtime scope, approval decisions, and structured events for agent-authored work. Context Engine inside Cosmos workflows supports repository-wide AI code review across 400,000+ files through semantic dependency graph analysis. Cosmos is generally available and included on all paid plans.
The following table maps each playbook stage to its corresponding Cosmos capability.
| Playbook stage | Governance capability |
|---|---|
| Inventory and provenance | Replayable run records preserve prompts, tool actions, and touched files |
| AI code audit | E2E Testing adds a real-infrastructure validation pass before merge; workflow automation shares context across code review and E2E validation |
| Guardrails | Scoped runtime boundaries define where agents run and what they can touch; teams enforce human-in-the-loop policies rather than treating them as advisory |
| Continuous governance | Experts respond to triggers, approved patterns stay in a registry, and every action emits a structured event |
| Compliance posture | SOC 2 Type II and ISO/IEC 42001 certified |
Frequently Asked Questions About AI-Generated Code Risk
These are the questions CTOs and platform leads ask when building governance programs for AI-assisted and agent-authored code.
Related Guides
Written by
Paula Hingel
Paula writes about the patterns that make AI coding agents actually work — spec-driven development, multi-agent orchestration, and the context engineering layer most teams skip. Her guides draw on real build examples and focus on what changes when you move from a single AI assistant to a full agentic codebase.