Evaluate code migration tools by how well they support the migration loop. That loop cycles through automated rewrite, per-shard validation, human review, and retry. Every large-scale port covered here shipped through that loop, and a single automated pass never finished any of them.
TL;DR
In the large-port cases covered here, automated code migration tools complete 70-97% of scoped files or changes and stall on the long tail of complex files where semantics, tests, or cross-repo dependencies break. Google, Airbnb, Amazon, and Meta shipped by wrapping tools in a validate-fail-retry loop with sharded changes, per-unit CI gates, error feedback, and human review before merge. Score loop mechanics before rewrite demos.
Why the Loop Matters More Than the Rewrite Rate
Engineering teams evaluating agentic migration tools face a documented 70-97% automation ceiling between demo output and shipped ports. The migration successes in this guide still required measured residual work after the automated pass finished. Airbnb finished ~100 files by hand after reaching 97%, Slack reported 20% manual intervention, and Google reported roughly a quarter of changes human-authored. In the Google, Airbnb, Amazon, and Meta cases below, shipped migrations depend on the same loop mechanics: sharded changes, per-unit CI gates, and human review on every shard from creation through merge. This guide maps the four categories of code migration tools, the automation ceilings each category hits, the loop mechanics that separate finished ports from stalled ones, and the evaluation criteria that predict which is which.
The Four Categories of Code Migration Tools
Code migration tools fall into rule-based AST rewriting, large-scale change management platforms, AI/LLM agentic rewriting, and framework-specific codemod collections. Each category automates a different slice of the port, and the documented results below show what each slice looks like at production scale.
| Category | Representative tools | Mechanism | Typical scope | Documented result |
|---|---|---|---|---|
| Rule-based / AST | OpenRewrite 8.87.2, jscodeshift, Codemod platform (JSSG), Putout | Deterministic Lossless Semantic Tree or AST recipes, not regex or AI heuristics | Java 8→21, Spring upgrades, CVE remediation, JS/TS codemods | Rule-based tools fit repetitive syntax, API, and dependency edits that deterministic recipes can express |
| Large-scale change management | Google Rosie (internal), Moderne (commercial) | Shards master changes into independently testable, reviewable, submittable units | Cross-repo recipe orchestration | Rosie lets large changes move through independently reviewable units across the Google codebase |
| AI / LLM agentic | Amazon Q Developer's Java upgrade workflow, AWS mainframe modernization service, GitHub Copilot App Modernization | Plan, rewrite, then build-and-fix iteration with a human review gate | Java version upgrades, .NET porting, COBOL-to-Java | Amazon Q Developer's Java upgrade path documents an analyze, rewrite, build, and fix workflow for Java application upgrades |
| Framework codemods | react-codemod, next-codemod, jest-codemods, ts-morph, 5to6-codemod | Framework-specific AST scripts | Framework API version bumps | Framework codemods automate framework API edits but do not provide cross-repo orchestration by themselves |
The Rosie sharding pattern breaks a master change into independently reviewable, submittable units across Google's codebase, and Amazon Q Developer's Java transformation workflow shows how the agentic category threads planning, rewriting, and build-and-fix iteration through a human review gate. Within the 2025-2026 examples named in this guide, the agentic category covers Java, .NET, and mainframe workloads, and the documented examples use assess/plan/rewrite/validate-style workflows. Those examples are GitHub Copilot App Modernization, AWS's mainframe modernization service, and Claude's AWS mainframe modernization plugin for COBOL-to-Java modernization. GitHub announced Copilot modernization GA for Java and .NET on September 22, 2025, with C++ support in private preview for Visual Studio 2026. AWS's mainframe modernization service still supports z/OS COBOL, PL/I, JCL, and CICS for existing customers, and offers both a deterministic Refactor path and a Reimagine path that reverse-engineers business rules into cloud-native services. AWS has closed the service to new customers: the Managed Runtime Environment experience stopped accepting new customers on November 7, 2025, and the Self-Managed Experience closes to new customers on June 30, 2026, with AWS pointing new mainframe workloads to AWS Transform for mainframe.
The Automation Ceiling: What Tools Complete on Their Own
Automated rewrite tools, whether rule-based codemods or LLM agents, complete 70-97% of the scoped work in the documented migrations below and leave a structured residual that requires human intervention. The ceiling appears across independently documented cases. Repetitive syntax and API changes automate well, while semantic intent, test translation, and cross-component behavior create the residual.
| Organization | Migration | Automation rate | Residual |
|---|---|---|---|
| Airbnb | Enzyme → React Testing Library, 3,500 files | 97% after four days of prompt tuning (initial pass: 75% in four hours) | ~100 files finished manually; long-tail files needed 50-100 retries vs. 10 for simple files |
| Slack | Enzyme → React Testing Library | 80% content accuracy | 20% required manual intervention |
| 39 internal LLM-assisted migrations over 12 months | 74.45% of code changes LLM-generated | Roughly a quarter of changes human-authored, with most of that work concentrated in complex interface and multi-component work |
Airbnb's engineering team documented its Enzyme port in a postmortem on large-scale test migration with LLMs, where long-tail files needed 50-100 retries against 10 for simple files. Slack's engineering team described a parallel effort in its writeup on Enzyme-to-RTL conversion, and Google's aggregate results across 39 internal migrations sit at the low end of the ceiling. Five failure modes explain the residual.
- The syntactic-semantic gap. Research on C-to-Rust migration found compiler feedback can repair syntactic and ownership violations, but logic errors where compiled code misbehaves demand deeper semantic understanding. A Java 8→11 LLM benchmark study using Mistral Codestral reached the same conclusion from the other direction: the model produced identical migrations in only 11.11% of cases and struggled with complex APIs, confirming that semantic migration failures dominate the tail regardless of whether the rewriter is rule-based or LLM-based.
- Test intent drift. Cross-library test migration tools often fail to preserve original test intent when semantics diverge; JTestMigrator managed a 73% success rate in Java even with semantic-similarity methods.
- No dynamic feedback. A 2025 survey of LLM migration agents names a dynamic feedback gap when agents lack automated setup, execution, and results-informed testing.
- Long-tail retry explosion. Airbnb's postmortem shows the effort curve inverts at the tail: the jump from 75% to 97% cost more calendar time than the first pass.
These failure modes explain why migration tools need validation and retry infrastructure alongside rewrite models.
Run your software agents at scale
Cosmos gives your agents the context, tools, and feedback loops they need to get better with every workflow.
The Migration Loop: How Ports Actually Ship
The migration loop converts automated output into merged production code through repeated discovery, code rewriting, CI validation, failure feedback, and human review. Files only advance when validation passes, and concrete error feedback moves files from automated output to merged code.
The named implementations differ in tooling but share the same shape.
- Google uses Kythe and Code Search for discovery, ClangMR or LLMs for code rewriting, and Rosie to shard the master change by project boundaries and ownership rules into atomically submittable units, each with its own test-mail-review-submit pipeline. The preferred pattern keeps every shard independently valid by introducing the new function, migrating all call sites, and then removing the old one.
- Airbnb structured its Enzyme port as a state machine. Each file moved through discrete gates (Enzyme refactor → Jest fixes → lint → TypeScript), with error messages feeding LLM retries automatically across hundreds of concurrent files. The original estimate was 1.5 years of engineering time; the loop shipped it in six weeks.
- Amazon Q Developer runs an analyze, rewrite, build-and-fix iteration, then holds the change until a human reviews the code with build and test results.
- Meta's CodemodService requires review of every automated diff without exception, and the company treats reviewer throughput as loop infrastructure. Meta's 2022 writeup on ML-based diff routing reported a 17% increase in review actions per day. Augment Code's automated PR analysis achieves a 59% F-score for code review quality on its AI code review benchmark, which checks pull requests against repository-wide context, architectural patterns, and team standards.
Platform teams built each of these four loops in-house. For teams without a dedicated migration platform group, the operational gap lies in applying those mechanics around the rewrite tool: sharding, event triggers, CI/CD integrations, and review-policy enforcement. Augment Cosmos, a unified cloud agents platform, runs agents in the cloud with shared context and memory that compound across the team and the software development lifecycle. It exposes three primitives (Environments, Experts, and Sessions) that a migration platform group would otherwise have to build internally, so teams running validate-fail-retry migration programs can pair those primitives with human-in-the-loop policies to manage synthesis, triage, implementation, testing, and review preparation.
The Criteria That Predict Whether a Tool Ships the Port
The criteria that predict migration success are structural properties of a tool's workflow. The named implementations above make those workflow properties visible. Vendor evaluations that omit those primitives do not test the documented shipping workflow.
| Criterion | What to evaluate |
|---|---|
| Semantic equivalence verification | Does the tool validate behavioral equivalence via differential testing or AST comparison, beyond compilation? |
| Per-unit validation | Can each shard pass CI independently before the next proceeds? A single end-to-end run at the end is incompatible with safe large-scale migration |
| Rollback granularity | Are outputs independently revertible units, or one atomic change? |
| Incremental decomposition | Does the tool shard by project and ownership boundaries so teams execute independently? |
| Long-tail handling | Does automation leave a usable partial baseline for manual completion, or a mess? |
| Discovery before code rewriting | Does the tool run dependency and blast-radius analysis before generating changes? |
| Ownership model fit | Does the workflow match monorepo vs. multi-repo and centralized vs. distributed ownership? |
Migration evaluations also need an organizational criterion most tool comparisons skip: who owns residual work after automation stops. Review-policy enforcement also depends on AI code review tools that check migrated shards against repository-wide context, architectural patterns, and team standards before reviewers see them.
The Validation Stack Behind the Loop
Semantic-equivalence validation verifies migrated behavior through characterization tests, dual-running, shadow traffic, and mutation testing before cutover. Each layer gates a different phase of the migration loop.
| Phase | Technique | Gate condition |
|---|---|---|
| Before touching code | Characterization tests / golden master | Team has captured a snapshot of current behavior, and it passes |
| During refactoring, offline | Teams refine approval tests into named unit tests | Golden master shrinks as teams name behaviors |
| During refactoring, online | GitHub Scientist runs old and new paths, returns the old, compares results | Mismatch rate drops to zero on production traffic |
| Pre-cutover | Diffy shadow traffic multicasts requests to both implementations | Candidate matches primary across a large, diverse request set |
| Test suite quality gate | Mutation testing | Mutation score meets threshold before cutover |
Each row in the table maps to a specific technique. Teams start with characterization tests that pin down current behavior before any code changes, then refine those approval tests into named unit tests as golden master coverage shrinks. During refactoring, GitHub Scientist runs old and new paths side by side on real traffic while still returning the old result, and Diffy shadow traffic multicasts pre-cutover requests to both implementations for large, diverse comparison sets. Two caveats surface in production use. Characterization tests only detect change; they do not identify whether the captured behavior was correct in the first place. Dual-running also doubles compute per request, so teams bound the duration and traffic share rather than running it indefinitely.
The Cross-Repo Context Gap
In the Sourcegraph product example below, AI agents fail migrations at the context boundary when the required dependency evidence sits outside the file or repository being edited. Sourcegraph describes this product limitation as the 80% problem, where agents complete visible edits but miss dependencies outside single-repository or file-local analysis: the auth middleware wrapping the changed function, DTOs serialized at another layer, and sibling repo tests that a single-repository agent never sees.
| Dimension | File-local migration boundary | Discovery-first migration loop |
|---|---|---|
| Failure boundary | Visible edits complete, but the agent misses dependencies outside the current file or repository | Whole-codebase discovery runs before code rewriting |
| Example dependency | Auth middleware wrapping the changed function | Cross-repo search surfaces downstream files |
| Serialized contract | DTOs serialized at another layer remain outside local analysis | Dependency mapping includes layers touched by the change |
| Test coverage gap | Sibling repo tests are not visible to a single-repository agent | Cross-repo tests become part of the validation surface |
| Migration-loop gate | The rewrite step can finish before CI or cross-repo search detects missing dependencies | Each shard advances only after CI, review, and context checks pass |
During migration-agent evaluation, Augment Code's 70.6% SWE-bench score is an initial benchmark result, and teams still need their own pilot-shard validation baseline before cross-repository CI becomes the merge gate. For the legacy languages discussed here, benchmark coverage is thinner because no major coding-agent benchmark evaluates COBOL, Fortran, or pre-modern Java at all.
Closing the cross-repo context gap requires whole-codebase discovery before code rewriting. In Sourcegraph's product example, an agent without full codebase context missed six downstream files, while cross-repo search surfaced all 31 references across seven layers and edited them as one unit. Google runs the same discovery-first architecture with Kythe and Code Search. Augment Code's Context Engine runs the same shape of analysis: it processes entire codebases across 400,000+ files through semantic dependency graph analysis, call flow analysis, and cross-repo semantic retrieval, exposing architectural dependencies before each shard runs. Teams applying cross-repo dependency mapping inside this workflow can see 5-10x task speed-up. Cosmos then keeps the loop running across days-long or weeks-long migration programs: tenant memory retains reviewer corrections, Experts subscribe to failing-CI or new-shard events, and Sessions preserve each run so the migration program stays auditable end to end.
Score Loop Mechanics Before You Shortlist a Single Tool
Migration vendors compete on rewrite rate, while the timelines in this guide are set by residual migration-loop work. A tool that converts 95% of files but emits one atomic change will lose to a tool that converts 80% into independently revertible, per-shard-validated units. Before shortlisting, engineering teams should run one candidate against a single bounded shard of the target repository, wire the output through CI as a hard gate, and rehearse the run repeatedly before cutover. Teams evaluating multi-file refactoring tools should score the loop mechanics before comparing rewrite demos.
FAQ
Related
Written by
Paula Hingel
Paula writes about the patterns that make AI coding agents actually work — spec-driven development, multi-agent orchestration, and the context engineering layer most teams skip. Her guides draw on real build examples and focus on what changes when you move from a single AI assistant to a full agentic codebase.