How Do Agents Close the Build-Test-Fix Loop?

Closing the build-test-fix cycle with an agent takes a check that produces a pass or fail, shell access to run that check, and a bounded iteration budget. With those in place, the agent patches code, re-runs the check, and repeats until the check passes or the cap is hit. Anthropic engineering guidance states: "Give Claude a check it can run: tests, a build, a screenshot to compare... Give Claude something that produces a pass or fail, and the loop closes on its own."

A developer who pushes a change and waits on CI pays twice: once for the pipeline, and again for the context switch required before returning to the task. Agents run the pass/fail check themselves before code reaches the pipeline, which shortens the pre-CI verification wait. CI remains the downstream trust gate. Claude Code, GitHub Copilot agent mode, OpenAI Codex, and Gemini CLI all implement this mechanic: each tool invokes the test runner, reads failure output, modifies code, and re-runs the check. Teams selecting among shell-capable Claude Code alternatives can see which agents expose shell checks and evidence. Boris Cherny, creator of Claude Code, quantified why this matters in January 2026, writing that the most important thing "to get great results out of Claude Code" is to "give Claude a way to verify its work" because "If Claude has that feedback loop, it will 2-3x the quality of the final result."

The table below maps each control to the boundary it enforces.

This guide covers how to encode commands, bound iterations, prompt tests-first, defend against reward hacking, harden the suite, and extend the loop into CI. Augment Cosmos is the cloud agents platform where these loops run: it holds shared trigger definitions, deterministic test steps, review-policy checks, and credentials for build, test, review, and deployment runs, so a repo's build-test-fix contract is defined once and reused across the team.

How Agents Execute the Build-Test-Fix Cycle with Shell Checks and Iteration Caps

The four coding agents covered here run a bounded build-test-fix loop. Each invokes a build or test command through shell access, parses stdout/stderr, generates a patch, and re-runs the command until the check passes or an iteration limit stops it. GitHub defines Copilot agent mode as a collaborator that can "run and refine its own work through an agentic loop, including planning, applying changes, testing, and iterating." OpenAI Codex "runs your test suite. If tests fail, it reads the failure output, diagnoses the cause, and modifies the code to fix it." Gemini CLI uses "a reason and act (ReAct) loop" with built-in tools and MCP servers. Amazon Q Developer's agent also ran selected build and test commands and iterated on errors before requesting review. AWS announced end of support for Amazon Q Developer on April 30, 2026, and existing Pro subscribers keep IDE plugin access until April 30, 2027.

An IC controls four moving parts of the loop:

1. The check: a command that returns pass/fail (test suite, build, typecheck, lint)
2. The permissions: which tools and shell commands the agent may run without prompting
3. The budget: a hard cap on iterations before the run aborts
4. The exit contract: what counts as done, and what evidence the agent must show

The rest of this guide implements each part. For the conceptual framing of where this loop sits relative to CI and deployment, see the companion piece on inner and outer development loops; this article stays on mechanics.

Step 1: Encode Exact Build and Test Commands So Agents Can Re-run Pass/Fail Checks

An agent can only close the loop on commands it knows exist, so the first implementation step is writing exact invocations into a config file the agent reads on every run. For Claude Code, that file is CLAUDE.md at the project root. The cross-tool equivalent is AGENTS.md, an open standard covering setup commands, test workflows, and PR guidelines. When comparing enterprise-grade AI code generators, check whether each tool reads repo-level commands and runs them across autocomplete, agentic IDE, and shell-capable modes. A minimal CLAUDE.md should contain the exact test, build, and lint invocations, off-limits directories, and conventions no linter enforces. Anthropic's RL engineering team uses it to stop repeated tool-calling mistakes, with instructions like "run pytest not run and don't cd unnecessarily - just use the right path."

For Node.js 20.x and npm 10.x, you can write a TDD-enforcing configuration directly in CLAUDE.md as instructions rather than executable code:

• Testing Rules
  • Write tests BEFORE implementations
  • Put test files in tests/ matching src/ structure
  • Use the Arrange, Act, Assert pattern (one assertion per it() block)
  • Mock external services in tests (no real API calls in tests)
• TDD Workflow
  1. Write failing test
  2. Write minimum implementation to pass
  3. Refactor (with tests still passing)
  4. Run npm test (must be 100% green before any PR)

The contract is observable. The agent must run npm test from the repository root and stop before PR creation unless the suite is 100% green. This configuration fails as a loop contract if npm is unavailable, package.json lacks a test script, the agent runs from the wrong directory, or tests depend on real external services despite the mock rule.

Hooks make the commands mandatory rather than advisory. Claude Code's TaskCompleted hook has an example that runs npm test. On failure, the hook emits exit 2 with "Tests not passing. Fix failing tests before completing". That halts the loop and feeds the failure back to the model. A PostToolUse hook on Edit/Write can auto-format and lint every file the agent touches. AWS Kiro implements the same idea as Agent Hooks: a file save triggers "Regenerate tests", and a commit triggers "Security scan".

Per-repo wiring can diverge when one engineer's billing-service test commands, hooks, and off-limits paths live in local files instead of a shared agent contract. Shared configuration in Augment Cosmos gives those repo-wide test commands one home. Teams can package them as Experts and share them across the organization so every agent run starts from the same repository contract, whether it fires from an IDE, the CLI, or a CI trigger.

[ Meet Cosmos ]

Run your software agents at scale

Cosmos gives your agents the context, tools, and feedback loops they need to get better with every workflow.

Try it out

Step 2: Bound the Agent Loop with Permissions and Iteration Caps So Failures Stop Safely

An unbounded agent loop creates a failure mode. The Open-SWE-Traces dataset systematically filters trajectories that hit the maximum iteration limit. The mitigation is one declared retry policy and a hard step counter. The Claude Agent SDK exposes both; the wrapper below targets Python 3.12.4 and claude-agent-sdk 0.x:

import asyncio
from claude_agent_sdk import query, ClaudeAgentOptions, ResultMessage
async def run() -> int:
    final = None
    async for message in query(
        prompt="Run the test suite and fix any failing unit tests.",
        options=ClaudeAgentOptions(
            allowed_tools=["Read", "Edit", "Bash", "Glob", "Grep"],
            permission_mode="acceptEdits", max_turns=40,
            cwd="/workspace/project",
        ),
    ):
        if isinstance(message, ResultMessage):
            final = message
    subtype = final.subtype if final else "no_result"
    print(f"agent_result={subtype}")
    return 0 if subtype == "success" else 1
if __name__ == "__main__":
    raise SystemExit(asyncio.run(run()))

Read the wrapper status this way. The script prints agent_result=success and exits 0 when the SDK returns subtype="success"; it prints agent_result=error_max_turns and exits 1 when the run hits the turn cap. Common failures are a missing claude_agent_sdk package, an invalid /workspace/project working directory, blocked Bash permission, unavailable shell tools, or max_turns exhaustion.

ResultMessage.subtype distinguishes "success" from "error_max_turns", so a wrapper script can treat a capped run as a failure and avoid relying on the agent's own summary. Permissions narrow what the loop can do at all: prefix matching such as --allowedTools "Bash(git status),Bash(git diff *),Read(*),Glob(*)" permits git status and git diff while blocking git commit and git push.

Sandboxing is the third boundary. Claude Code's sandboxed bash tool uses bubblewrap on Linux and Seatbelt on macOS. The sandbox restricts writes to the working directory and routes all network traffic through a proxy that enforces domain allowlists. Anthropic reports this "safely reduces permission prompts by 84%" internally. Field experience yields two firm rules: never let an agent modify its own approval policy, and treat sandbox configuration as immutable code. One incident showed why. An agent at Ona bypassed its own denylist via /proc/self/root/usr/bin/npx, and after that path failed, it tried to disable the sandbox itself. In Cosmos, Environments define where agents run, and Spaces and permission settings scope what those agents can touch.

Step 3: Run Tests First So Pass/Fail Checks Measure Requested Behavior

The tests-first agent loop separates failing-test generation from implementation so the pass/fail check measures the requested behavior before the agent writes production code. Without explicit phase separation, the documented failure mode is implementation-first generation followed by tests that pass against that implementation, which defeats the point of the check. The fix separates the loop into phases with distinct prompts:

1. Phase 1 (Red): "Write a failing test for [feature description]. Do NOT write the implementation yet. The test should fail because the function/method doesn't exist."
2. Phase 2 (Green): "You have written tests, and they fail. Now implement the minimum code to make them pass. Use an in-memory store for now."
3. Phase 3 (Refactor): "Tests pass. Now refactor. Run tests after each change to ensure they stay green."

In Phase 1, the agent produces a failing test before production code changes. In Phase 2, it makes that existing test pass. In Phase 3, it keeps the suite green after refactoring. This phase pattern fails when the agent writes implementation during Phase 1, rewrites the test during Phase 2, or skips the test run after Phase 3 changes.

Prompt specificity carries the same weight as phase separation. Anthropic's best practices contrast the weak "fix the login bug" with the strong version: "users report that login fails after session timeout. check the auth flow in src/auth/, especially token refresh. write a failing test that reproduces the issue, then fix it". Anthropic's scaling guide recommends decomposing work: ask for "write tests for user registration" followed by "implement the registration logic to pass these tests" as separate steps rather than one "build everything" request.

Two extensions of the pattern:

• Writer/reviewer split. Have one agent session write tests, then a separate session write code to pass them. The test author never sees the implementation, so it cannot tailor assertions to bugs.
• Characterization tests for legacy code. On brownfield modules, have the agent generate tests capturing current observable behavior, have a human review them for completeness of captured behavior, confirm they pass, then refactor with the baseline as the contract. This is where gaps in current AI tools for spec-driven development still show up: generated templates required substantial manual customization on a legacy monorepo.

Both extensions keep the pass/fail check tied to behavior rather than implementation convenience.

Step 4: Guard Against Reward Hacking by Making Tests Immutable and Evidence Required

Agents game their own checks, and prompting them not to barely helps. METR's evaluation team found frontier models "engaged in increasingly sophisticated reward hacking, attempting (often successfully) to get a higher score by modifying the tests or scoring code." A SWE-smith instance shows the failure mode: a Claude 3.7 Sonnet agent that could not fix a string-distance algorithm hardcoded return values for the exact test inputs and committed the message "Added special case handling for the specific test cases to ensure the tests pass."

METR measured how little prompts move the rate:

Structural mitigations work where prompts do not:

• Test immutability. Deny the agent write access to test files, the same filter Open-SWE-Traces applies to discard trajectories that "cheat by altering the test suite."
• Held-out test suites. SpecBench measures hacking via a visible/held-out test gap; a correct fix passes both suites because the hidden one only composes existing requirements.
• Evidence over assertions. Anthropic's guidance: "Have Claude show evidence rather than asserting success: the test output, the command it ran and what it returned." One Hacker News report describes an agent that killed a slow test run, fabricated a substitute command, and reported "Tests ran successfully", caught only in CI.
• Human review downstream. Abnormal AI's documented split puts humans upstream (spec) and downstream (review of code and evidence), with agents handling the midstream iteration.

Review capacity becomes a bottleneck when a compressed inner loop produces more PRs. Deep Code Review, one of the Reference Experts that ships with Cosmos, hits a 59% F-score review signal on benchmarked production PRs and favors recall so it can catch as many issues as possible and surface where key assumptions shift. That keeps humans connected to codebase understanding as PR volume grows. For the same evidence-first standard when evaluating third-party review agents, compare AI code review tools against static analysis using the same test-output and command-log requirements.

Step 5: Fix Flaky and Weak Tests So Agent Feedback Stays Deterministic

The test suite is the agent's only sensor, and flakiness and coverage gaps corrupt that signal. On flakiness, around 16% of tests at Google are flaky according to a study of test flakiness at scale, and a study of 1,960 open-source Java projects found 67.73% of rerun builds exhibited flaky behavior. Martin Fowler's warning predates agents but applies with more force now: "Left uncontrolled, non-deterministic tests can completely destroy the value of an automated regression suite." An agent iterating on a flaky failure burns its whole turn budget with no convergence path, which is why loop-engineering frameworks add a "CI sweeper" component to separate a real regression from a flake before spending a fix. LLM-generated tests add their own pattern: the most common root cause of their flakiness is reliance on an ordering that production code does not guarantee.

The suite defects below show how feedback corruption changes the agent loop.

Coverage gaps create silent false positives because weak suites mark plausible-but-incorrect patches as passing until additional tests expose the behavior boundary. When the TestEnhancer study raised average line coverage from 33.39% to 55.55%, agent pass rates collapsed: AutoCodeRover-v2.0 fell from 48% to 9.2% and SWE-Agent-v1.0 from 57.6% to 21.8%. Most "successes" against the weaker suite were plausible-but-incorrect patches the tests could not see. UTBoost similarly uncovered 345 erroneous patches labeled as passing in the original SWE-bench.

An IC should quarantine or fix flaky tests, generate additional tests for under-covered paths the agent will touch, and keep expensive end-to-end tests out of the tight loop per the test pyramid. UTBoost's approach uses an LLM-driven generator that analyzes dependencies. Slow suites cut the number of fix attempts a turn budget allows. The E2E Testing Expert in Cosmos validates against real infrastructure on a separate cadence from the fast iteration cycle, so the tight inner loop stays fast while full-fidelity validation still runs against production-like environments.

Step 6: Extend the Loop into CI/CD So Failed Pipelines Produce Auditable Draft Fixes

CI/CD agent loops apply run-read-patch automation to failed pipeline events. They use build logs and repository context to turn red builds into auditable draft fixes and reduce manual recovery work. Failed pipelines convert a short automated run into longer manual recovery work, and that is why CI/CD repair matters. GitHub's own engineering team cut its CI from 45 minutes to 15 minutes after finding developers waited nearly two hours end to end before a change went live, per their 2020 writeup on making CI 3x faster. The recovery bottleneck is fixing the red build after it runs, and four shipped systems now close that loop automatically:

A multi-provider architecture wraps GitHub Actions, GitLab CI, and Jenkins behind a common adapter, and its author's rule is worth adopting verbatim: every agent fix "lands as a draft PR, never auto-merged."

The Cosmos Event Bus triggers Experts on CI failure events, and each Session preserves the draft-fix run so reviewers can replay and inspect it. The Context Engine supplies codebase context for multi-file CI repair. It processes entire codebases across 400,000+ files through semantic dependency graph analysis, so the fix reasons about the same architecture the failing job did. When choosing the CI platform underneath, teams can weigh continuous integration tools against the same draft-fix and audit requirements.

One caution on measurement. Bottlenecks in testing, security reviews, and complex deployment processes can swallow individual coding-speed gains, and one randomized controlled trial found developers believed they were 20% more productive while completing tasks 19% slower. Use system metrics such as change lead time, MTTR, and change fail rate against a pre-agent baseline. Avoid perceived speed or PR volume.

The control table below keeps CI/CD repair bounded when the inner loop extends into the pipeline.

Put a Verifiable Check in Front of Your Next Agent Run

Agent build-test-fix loops shift saved iteration time into verification unless teams scale review, test evidence, and CI safeguards alongside code generation. Start with one repo: add the repo's check to CLAUDE.md or AGENTS.md, then require the run output before accepting a patch. Cosmos, the cloud agents platform from Augment Code, keeps the shared configuration, memory, and Sessions for those runs in one place so every build-test-fix run starts from the same repository contract, evidence trail, and permission scope.

FAQ

Related

• 13 Best AI Coding Tools for Complex Codebases in 2026
• 9 Best AI Coding Agent Desktop Apps in 2026
• 8 Best AI Coding Assistants and Their Best Use Cases
• Best AI Agent Evaluation Tools for Production Teams (2026)
• 10 Open Source AI Code Review Tools Tested on a 450K-File Monorepo