The engineering manager artificial intelligence (AI) job is supervising by exception because written gates route risky agent output to human judgment.
TL;DR
Reports include agents and the humans gating them, so a manager who reads every diff becomes the slowest station on the line. Supervising by exception replaces that habit: the manager samples agent runs and signs off on halts. The role records postmortem and promotion evidence, then makes transition decisions around baselines and operating cadence. The manager calibrates thresholds from sampled records.
An engineering manager (EM) whose team runs a fleet of coding agents still reads every pull request (PR) the fleet opens and still sits in the approval chain for every merge. The fleet opens changes faster than one person can read, so the queue lengthens and each approval gets shorter. The manager becomes the one station on the line that cannot add capacity. By Monday, the queue is longer than it was on Friday. Supervising by exception is how the manager owns the gates; agent output must clear without reading every diff first.
This guide is for the EM whose reports include agents and the humans who gate them, and for the chief technology officer (CTO) or Vice President of Engineering deciding what that manager's job should be. It stays at the level of one manager's practice. Role definitions and gate ownership across squads and platform teams are org-level software factory design decisions. Change-management and adoption programs are out of scope.
What the Engineering Manager AI Job Owns
The EM in a software factory owns the exception list and gate system, including authorship delegation. The gates are the points where agent output waits for a person. Authorship delegation sets which reports may direct agents against which parts of the codebase; the human report signs the spec and owns the result while the agent writes the code.
A specification owner writes the bounded task, a verification owner checks the result, an agent operations owner keeps agents running, a service owner holds the authority to halt the workflow, and a gate owner signs a specific gate. The org design guide defines those roles, and the software factory model sets out the line layout.
New work concentrates in requirements, coordination, and deployment as implementation gets cheaper, plus reviewing generated code and evaluating agent behavior, according to a 2026 practitioner interview study preprint built on 20 interviews across 12 organizations, supplemented by a broader survey. Each is a gate or an exception, and none is a ticket to assign.
Cosmos, Augment Code's unified cloud agents platform available on all paid plans, gives the EM records that connect each agent run to its gate signal and accountable approver.
Supervising by Exception
Supervising by exception means the EM writes down, in advance, which signals reach a human and at what threshold, then samples the rest and acts only on deviations from that written standard. NIST AI 600-1, the National Institute of Standards and Technology (NIST) generative AI profile, recommends in action GV-1.3-002 that organizations set minimum performance thresholds as part of go/no-go deployment approval and leaves the values to the organization. Diff Risk Score thresholds vary by the source of the change in Meta's RADAR automated review system, described in a 2026 preprint, with human-authored diffs held to a stricter threshold than non-allowlisted AI or bot sources.
Blast radius and reversibility decide what the agents absorb and what reaches the EM.
| Signal | Evidence surface | Handled by | Reaches the EM when |
|---|---|---|---|
| Inline review finding on a routine change | Automated inline comments on the PR | Specification owner fixes and re-runs | The same finding recurs across sampled run records |
| Blast radius or security exposure flag | Risk analysis on the PR | Gate owner review | The flag touches auth, payments, data migration, or a shared interface |
| End-to-end check failure | Output from the verification run | Specification owner re-runs | The same failure repeats on a second attempt |
| Error budget burned past policy | Alert from the monitoring system | Service owner halts | Every halt, before sign-off |
For a flagged change, the EM reopens the Cosmos Session, the retained and reopenable record of everything the agent did during that run, and reads the agent's steps against the spec rather than reading the diff alone. Reviewers agreed on risk only moderately, with a Fleiss' kappa of 0.52, across 125 adversarially weighted agent actions in an oversight-capacity study preprint. In the study's reviewer-fatigue model, realized safety follows an inverted U as escalation load grows, so escalating past a threshold lowers safety. The Risk Analyzer Expert in Cosmos surfaces the blast radius, and security exposure flags the EM reviews. After reviewing sampled records and classifying an alert as a false positive, the EM raises a signal-class threshold. The EM lowers it when that review finds a defect with no signal raised.
Why the Manager's Job Changes Now
Managers who keep reading every diff lose the review anyway, because coverage falls as volume rises. Per-capita PR throughput reached 2.09 times the pre-mandate baseline across 802 developers and 196,212 PRs, per a preprint on an enterprise PR mandate to double output. The share of PRs with at least one human review fell from 89% to 68%. Human review became a sample, whether or not anyone designed it as one.
Google Cloud's DORA 2025 report describes time saved during initial code generation as often re-allocated to verification overhead, with 30% of respondents reporting little or no trust in AI-generated code. Direction and evaluation of AI output, including correction, formed a category named "supervisory engineering work" in a matched cohort preprint of 95 participants. How much of that work one manager can carry is a planning question. The EM budgets supervision bandwidth beside agent compute in engineering capacity planning.
Staffing Your Own Week: Span of Control Over a Fleet
One manager faces smaller questions: which agent runs to open and how many humans to supervise. The org-level team topology decision, which settles whether the experts on a line sit with a platform team or with the squad, is out of scope here. A separate question is what a 1:1 covers when an agent produced a report's output.
The EM makes a written commitment on Monday to a personal sample of recorded Cosmos Sessions behind the fleet's merges, because a sample that flexes with the calendar shrinks first. It covers every run behind a halt before the EM signs that halt off and every run behind a change the Risk Analyzer flags for blast radius or security exposure before the gate owner's approval stands. The EM randomly draws a fixed count of unflagged runs and sets that as the starting number. The EM then tunes it against what the sample reveals. On a line with more agent-authored merges, the number increases until the sampling hours no longer fit the week, at which point the span stops.
The line's human-agent ratio measures agents per human separately from the EM's sampling limit. A weekly 1:1 with a specification owner or verification owner whose output came from an agent drops status and asks about direction and evaluation, including correction.
- Runs opened: The report names which run records it opened beyond the ones that failed, and what changed as a result.
- Exceptions absorbed: The report explains which agent findings it accepted without escalating, and why those sat within its authority.
- Spec revisions: The report names which specs it rewrote after an agent misread them, and which boundary was missing.
- Verification debt: The report lists which merged changes the verifier has not yet exercised end to end.
A verification owner who cannot answer the verification-debt question leaves work that the EM's own sample of Cosmos Sessions has to absorb, which signals the span is too wide.
Halt Authority and Incident Ownership in Practice
Org-level policy assigns stop, escalation, and restart rights for a line and ties incident ownership to the service owner who holds them. The EM's work begins when a service owner has already pulled the cord.
Before signing off a halt, the EM opens the Cosmos Sessions behind the triggering signal and reads them against the written exception list. Google's site reliability engineering (SRE) workbook error-budget policy applies when a service has exceeded its error budget for the preceding four-week window, and halts "all changes and releases other than P0 issues or security fixes." The EM confirms the halted line lets only that class through, and nothing else. The same policy calls for a postmortem whenever a single incident consumes more than 20% of the four-week budget.
The 1:1 the week after a drill covers elapsed time from signal to halt and whether the owner waited for anyone before pulling. A service owner who reports checking with the EM before halting has learned that halting needs permission, and the EM retracts that lesson on the record.
The EM writes the ownership and control record into the postmortem personally.
- Ownership line: The postmortem names one person as the single point of contact for the postmortem; the model Google's SRE postmortem culture chapter ties to action getting done, with each action item inside it carrying its own named owner.
- Control record: The entry states the exception threshold, when the EM last tuned it, whether the halt was a true positive, and which Cosmos Sessions in the affected line no one had opened before the incident.
A postmortem that names the sampling gap tells whoever inherits the line which run records to open first.
Where This Goes Wrong
Supervising by exception fails in three familiar shapes, and each one still looks like diligence from the outside.
- The EM as bottleneck: The manager keeps a personal approval on every merge. In the enterprise PR mandate study, substantive human review fell from about 39% to about 21% of PRs. Remove your approval from routine gates this week and keep it on the exception classes in the table above. Log which run records you opened in its place.
- The EM as a rubber stamp: The manager still reviews, and the reviews get lighter. Approval rates rose from 30.1% to 36.8% as reviewers gained experience with agent PRs, according to a reviewer habituation study preprint that analyzed reviews by 400 repeat reviewers in the AIDev dataset. Le and Kunz's AI complacency experiments in the Journal of Service Management, six studies with 1,370 participants, found the primary driver of AI complacency was lack of accountability for monitoring AI output. Put your name on the sampled run records in writing. When your own change-request rate on the sample falls, cut your span.
- The EM absent from the halt drill: The manager approves the drill plan and does not attend. In an agent sabotage study preprint, 94% of more than 100 developers working beside a covertly sabotaging coding agent for about five hours in a coding task failed to detect it. Attend the next drill and time it from signal to halt. Write that duration into the postmortem yourself.
The correction in all three is the exception list and the run records that show the EM read it. The same run records settle who gets promoted next.
Hiring, Leveling, and Career Design Under Delegated Authorship
Organizations re-price interviews and ladders under delegated authorship at the org-policy level. The EM's part is the evidence that goes into a packet and the questions asked in a review.
A promotion packet cites the agent run records the candidate directed, where the spec survived agent execution without revision. It records the exceptions the candidate caught that the exception list had missed. It includes the halts the candidate called that a postmortem later confirmed. The DORA metrics guide warns against measuring complex systems on any single metric, so a packet built on merged-PR volume in a fleet where agents author most PRs is measuring the fleet. The performance review asks what the report delegated and what human judgment they exercised, including personal verification and stop decisions, with one run record as evidence under each. Sonar's State of Code survey of 1,149 professional developers found 47% named reviewing and validating AI-generated code for quality and security as the most important skill in the AI era.
From Junior to Senior, a paper at the Association for Computing Machinery (ACM) Conference on Human Factors in Computing Systems (CHI) 2026, found that seniors infer junior competence and thinking patterns from prompt quality and specificity. The EM reads the same way, with the spec beside the agent run record it produced. The spec has to name the task boundary and the verification criterion. The record has to show the agent working inside them.
Junior engineers in the AI-assisted arm of a randomized trial preprint scored lower on a post-task quiz, with the widest gap on debugging questions. The EM asks the junior to explain a failing case in the run record before accepting the agent's fix.
Transition Decisions From Assisted to Assembly Line
The software factory maturity model places Assisted before Assembly Line, and the gap between them is the gap between reading every diff and owning the gates. Manager adoption establishes the first decision area: whether the EM has enough direct experience to define a review-time baseline. Engineers whose manager used Copilot's command-line interface (CLI) had 82% higher odds of trying it themselves and 22% higher odds of still using it after two weeks, according to a Microsoft internal study that tracked telemetry from tens of thousands of engineers during an early-2026 Copilot CLI rollout, supplemented by a 609-response survey used to interpret the findings. That study measures adoption, not supervision quality. The EM's own Cosmos Sessions provide the practice baseline: daily review preserves line-by-line inspection long enough to measure the time it consumes, while a draft exception list gives one line a threshold and an owner beside every signal.
A durable operating cadence requires the EM to reserve time for the halt drill and written weekly sample before other calendar demands displace them. The halt signal routes directly to the person who owns the response, so escalation doesn't wait on the manager. The EM attends the halt drill and replaces status reporting in 1:1s with the four evidence questions above.
Threshold calibration depends on what the EM finds in sampled Sessions, where the Deep Reviewer Expert in Cosmos records the findings behind each flagged change. The EM loosens thresholds only after reviewing those sampled records and classifying alerts as false positives, using the same source-tiered logic that set the thresholds in the first place: human-authored diffs held to a stricter bar than agent-authored ones. The manager writes the drill postmortem and drafts one promotion packet from run records. The manager then compares sampling time with the original review baseline. If the sample slips, the span has reached its limit, and the next hire is a verification owner.
What to Do Next
The tradeoff is supervision bandwidth against review depth, measured in the hours you spend reopening an agent run end to end and reading it against the spec that produced it. Every run you open fully is one you cannot open somewhere else, and a sample drawn to look thorough covers less of the line than a smaller one you commit to on Monday and reopen on Friday. Choose the line with the most agent-authored merges last month, and before Friday write down which of its run records you will personally open next week, with the signal and threshold beside each one.
Frequently Asked Questions About Engineering Manager AI
Related Reading
Written by
Ani Galstian
Ani writes about enterprise-scale AI coding tool evaluation, agentic development security, and the operational patterns that make AI agents reliable in production. His guides cover topics like AGENTS.md context files, spec-as-source-of-truth workflows, and how engineering teams should assess AI coding tools across dimensions like auditability and security compliance