Engineering leaders should measure AI code quality through change failure rate, rollback rate, escaped defects, and fresh-code rework because those metrics record whether a change survives review and production.
TL;DR
Cheap authoring broke the assumption every metric rested on: that a human wrote each line and writing was the expensive part. Line and commit counts now tally agent invocations. AI code quality shows after a change leaves review, in change failure rate, revert rate, defect escape, rework on fresh code, review findings per change, and cost per merged change.
A review rotation sized for six pull requests a day meets thirty overnight, and the delivery dashboard still reads healthy: throughput up, story points up, lead time flat. The rollback log tells a different story, because the dashboard counts what agents produced and the rollback log counts what did not hold. The board asks whether quality is holding, and every number on the slide was built to answer a different question.
This guide is for the Chief Technology Officer (CTO) and the Vice President (VP) of Engineering whose teams already run coding agents and who now have to decide which numbers deserve a slide. For autonomous pull requests, pull request review is the first shared human gate, and much of the argument sits there. Each measurement here answers one decision: how much agents ship unsupervised, where a gate needs staffing, and whether agent spend is buying output that survives. Linter configuration and dashboard construction stay out of scope; what a leader funds and asks the platform team for does not.
Why the Denominator Moved
Every conventional quality proxy assumed a person wrote each line and that writing was the scarce step. Lines of code approximated effort, while a commit approximated a considered unit of work. Velocity approximated how much a team could build in a sprint. All three measured the human cost of production and let quality ride along as an inference. Cheap authoring breaks that inference because the cost term those proxies tracked has collapsed.
At one enterprise under a throughput mandate, AI-authored pull requests went from almost none to roughly 90% of everything merged. Per-capita output reached 2.09 times the pre-mandate baseline over the 28 months tracked by the enterprise pull request study, a preprint. At that share the denominator under every per-line and per-commit metric is agent output. A team whose merged pull requests doubled did not double its engineering judgment, so engineering leaders must read quality where cost still lives: in review, in verification, in rollback, and in the comprehension work that reading an agent's change requires.
A proxy earns its place by standing in for something expensive, and the expensive thing is now the reading, the verifying, and the repairing. Any measurement taken at the moment of authoring now describes the cheap half of the work. That is why a healthy authoring dashboard and a deteriorating codebase are no longer in contradiction.
The Metrics That Became Actively Misleading
Line and commit counts can now move opposite to quality, because each rewards a behavior agents make free. Complexity, coverage, and duplication keep their uses, and the conventional definitions do not change with who wrote the line. The broken proxies all count acts, and none of them counts an outcome.
- Lines of code: A count that once implied hours of thought now implies a prompt. Faros AI, an engineering analytics vendor, found in its telemetry study of more than a thousand teams that in the quarters a team leaned on AI hardest, its pull requests ran 154% larger and took 91% longer to review. Line volume now reports prompt volume, which belongs in a capacity forecast and not a quality review.
- Commit count: Tooling that commits after each edit turns the commit log into an invocation log. Across millions of changes at one enterprise, a Google study found AI-generated changes drew 1.92 times as many blocking review threads as human-written ones, plus more failed builds and more sanitizer findings. Every one of those is an unadjusted contrast in its C++ sample. A doubled commit count says how many times an agent ran and nothing about what it left behind.
- Velocity: Story velocity counts points per sprint; cycle time counts hours from ticket to merge. Bugs inside pull requests rose 41% among developers who had GitHub Copilot access, against a matched group who did not, in a comparison Uplevel ran across roughly 800 developers on large teams. Cycle time and throughput moved neither way. The groups were formed by tool access, not measured usage, which prices availability and not adoption. Velocity measures how fast work enters the review queue, and the queue is where the constraint now sits.
Faros found none of its own team-level movement surviving aggregation: no significant correlation between AI adoption and company-level improvement, across overall throughput, DevOps Research and Assessment (DORA) metrics, or quality indicators.
What Agent-Authored Code Does to a Codebase Over Time
Generating a near-copy now costs less than finding the function that already does the job, and tracked codebases show both halves of that trade. In GitClear's maintainability report, duplicated blocks of five or more consecutive repeated meaningful lines rose from 40.3 per million changed lines in 2023 to 73.0 so far in 2026. Moved or refactored lines fell from 13% of changed lines to 3.8% across the same years. GitClear sells the measurement it reports on, and these are codebase-wide trends across the years agent authorship scaled, not agent-written lines measured against human-written ones. Duplication and complexity are visible before merge, rework and instability only after, and authoring-volume metrics catch neither. The same vendor treats two-week churn, lines rewritten or removed within two weeks of being authored, as a proxy for defect or rework rate.
Complexity is the weaker signal, and a dashboard built on it reads flat while the codebase degrades. ScenEval, a peer-reviewed Java benchmark, put generated and human reference solutions within a fraction of a point of each other on both cyclomatic and cognitive complexity, with generated code above the reference on slightly under half its tasks.
Rework is where the author comparison exists. Tracking 182 open-source repositories for thirteen months, the maintenance preprint found overall maintenance rates similar for agentic and human contributions, with corrective maintenance 46% higher on the agentic side. It also found the association a leader can act on: each 10 percentage point rise in a project's no-review rate, across all its merged pull requests, went with roughly a 6% rise in agentic maintenance burden.
Autonomous Agents and Autocomplete Carry Different Risk
An autonomous agent opening a whole pull request and an autocomplete assistant proposing one line carry different defect profiles, and a dashboard column labeled "AI-assisted" collapses both into one number. Each accepted autocomplete suggestion puts a human decision at the moment of insertion. An autonomous agent moves that decision to the pull request, after hundreds of lines and its own tests have shaped the change.
The security evidence splits the same way. The "Asleep at the Keyboard" study, peer reviewed at the IEEE Symposium on Security and Privacy, prompted Copilot across CWE Top 25 scenarios and found roughly 40% of its 1,689 programs vulnerable under CodeQL plus manual inspection. Those are small completions, one accepted decision each. Move up to whole changes and the picture concentrates. In the SusVibes benchmark, accepted at ICML 2026, its best setup, SWE-agent with Claude 4 Sonnet, left a vulnerability in 79.3% of the repository-level tasks it solved correctly. Different benchmarks and different methods: the pair sets no head-to-head rate, only a direction to look in.
| Dimension | Autonomous agent | Autocomplete assistant |
|---|---|---|
| Unit of change | Whole pull request or task | Single suggestion, line, or block |
| First human decision | At review, once the change is complete | At each accepted suggestion, during insertion |
| Where defects surface | Review findings and post-merge events, reverts and maintenance included | Logic and security flaws inside a human-shaped change |
| What a generic "AI %" tag records | The whole change as agent output | Accepted suggestions as agent output |
| Where the measurement is taken | At the gate and after merge, by authorship cohort | At the gate, inside human-authored changes |
A tag that records AI-committed code without the mode collapses both columns into one number, so the authorship tag has to carry interaction mode. Throughput and stability move independently often enough that one delivery number hides both, which is the mechanism behind the AI productivity paradox.
The AI Code Quality Measurements That Still Mean Something
The measurements that survive are taken where a human or a production system accepts or rejects the change: at the review gate, and in the weeks after merge. DORA's metrics guide, last updated January 5, 2026, defines change fail rate as "the ratio of deployments that require immediate intervention following a deployment" and pairs it with deployment rework rate to measure instability. It recommends those metrics and requires none of them. One of them keeps coming back flat. The Google study put reverts of AI-generated changes at roughly 0.9 times the human rate, and the enterprise under the throughput mandate held merge and revert rates steady while AI authorship climbed to 90%. Revert rate is a coarse, short-horizon proxy: a flat line there means read the gate measurements beside it, not that nothing changed.
| Measurement | What It Detects | Why It Survives Agent Authoring | When It Moves | Decision It Supports |
|---|---|---|---|---|
| Change failure rate | Deployments needing immediate intervention | The deployment system logs the intervention whoever wrote the change | Per deployment | Whether to widen or narrow what agents ship unsupervised |
| Revert and rollback rate | Merged changes withdrawn | The repository logs the withdrawal whoever wrote the change | Days after merge | Whether review depth matches change risk |
| Defect escape rate past each gate | Defects that got past the gate meant to catch them | Attributes the miss to a gate, not to an author | After the next gate or incident | Where to add or staff a gate |
| Rework on recently merged code | Lines changed again within two weeks of merge | Counts changes to code already merged, whoever authored it | Two weeks after merge | Whether agent tasks are scoped too large |
| Review findings per change and share addressed | Defects caught before merge and how many get fixed | Every autonomous change reaches review before it reaches production | Per change | Whether reviewers are a bottleneck or a rubber stamp |
| Time from defect introduction to detection | Delay between the causing commit and the finding | Times the gate's latency, not the author's speed | Per defect | Whether to fund earlier detection or faster rollback |
| Cost per merged change | Model, compute, and reviewer cost per merge that held | It prices review and rollback alongside authoring | Per billing period | Whether agent spend is buying durable output |
Of everything in that table, review capacity binds first, because reviewer hours do not scale with what agents can open. Across a few hundred Claude Code pull requests in open-source projects, an empirical study found 45.1% of the merged ones were modified on the way in, most often to fix a functional bug.
Once it binds, the decision is which changes still need a person, and that is a sorting problem before a hiring one. Cosmos, a unified cloud agents platform from Augment Code available on all paid plans, sorts that queue. The Cosmos Risk Analyzer auto-approves changes it judges low-risk, with a short justification, and flags the rest with a note naming what needs human attention; a Cosmos approval still leaves the merge to a person. The Cosmos Deep Reviewer reads the diff against repository guidance and posts inline findings. Both change what findings per change counts, because model and human findings land in the same thread and only one kind consumes reviewer hours.
Deterministic Checks on Non-Deterministic Output
With the analyzer version, ruleset, dependencies, and environment unchanged, a static analysis rule produces the same finding on the same code every run. A generative agent does not, so the enforcing layer has to be the deterministic one. A quality gate is a control that blocks a change on a fixed rule; the measurements here are instruments recording how often the rule fired and what got past it.
Of 302,579 AI-authored commits in an empirical study of the wild, 9.1% introduced an issue, and 22.7% of those issues were still sitting at the repository head. Counted per assistant rather than pooled, the same study puts that rate above 15% for every tool. Veracode, a commercial application security vendor that sells the scanning behind the finding, sent the same 80 coding tasks to more than 100 models for its code security benchmark. The average pass rate was 55%, with Java trailing the field at 28.50% against 61.69% in Python.
A Cosmos Expert such as Deep Reviewer produces findings from a model reading the diff, and that output varies between runs on identical code. A metric built on it measures the reviewer as well as the code. Where a path must never be auto-approved at all, Risk Analyzer setup takes repository allowlists and the paths that always require human review. Cosmos Environments, where Experts run, pause after inactivity and run for at most 24 hours. Between that and the run-to-run variation, a rule that must hold on every change belongs in the gate and not in an Expert.
What to Put in Front of a Board
A board acts on change failure rate as business risk, cost per merged change as unit economics, and security and compliance exposure as liability. All three need a trend, a denominator, and the boundary of the data behind them, and none needs a scorecard.
The 2025 DORA report, from a survey of nearly 5,000 technology professionals, finds that AI adoption now improves software delivery throughput, a shift from the previous year, and still increases delivery instability. A board that funded agent adoption should expect pressure on change failure rate and ask for the trend by authorship cohort, not a blended figure. Cost per merged change divides model usage, Cosmos compute, and reviewer time by the merges that held, with the revert log supplying the denominator.
At least one security smell turned up in 38.9% of the agent-authored pull requests scanned by a security-smell study accepted at the KDD 2026 Workshop on Agentic Software Engineering. All but one of its 253 critical findings was a hard-coded credential. The same study attributes 67.6% of the genuine leaked secrets to the human collaborators in those workflows, so the exposure belongs to the review process and not to the agent alone. Its detector is a model rather than a scanner, and what it counts are structural smells and not confirmed exploitable vulnerabilities.
The European Union's AI Act has applied since August 2, 2026, and for high-risk systems it requires human oversight, a risk management system, and record keeping. Coding assistants are not an Annex III category, so the exposure runs through what a team builds with them and not through the tools.
ISO/IEC 42001:2023 specifies requirements for an artificial intelligence management system, for organizations developing, providing, or using AI. Augment holds that certification and a SOC 2 Type II attestation, both on its security and compliance page; the ISO/IEC 42001 compliance features and customer-managed encryption keys sit on the Enterprise plan. Cosmos Sessions keep each run's conversation and work, saved indefinitely and reopenable. That is the record an auditor asks for when the question is what an agent did and on whose instruction.
What to Do Next
Measuring at the gate and after merge costs attribution work that a velocity dashboard was never built to carry. Every merged change needs an authorship tag, every incident tracing back to a change, and every escaped defect assigning to the gate that missed it. That plumbing competes with the agent rollout itself for the platform team's time, and it is worth one release cycle before deciding whether to keep it.
The platform team must join deployment, incident, and rollback records. Tag every change merged in the next cycle as agent-authored or human-authored, then report change failure rate and two-week rework for both cohorts side by side at the next board meeting, with each cohort's size beside its rate.
Frequently Asked Questions
Related Reading
Written by
Ani Galstian
Ani writes about enterprise-scale AI coding tool evaluation, agentic development security, and the operational patterns that make AI agents reliable in production. His guides cover topics like AGENTS.md context files, spec-as-source-of-truth workflows, and how engineering teams should assess AI coding tools across dimensions like auditability and security compliance