Software factory metrics treat the agent fleet and its human reviewers as one production unit. They measure engineering throughput, delivery predictability, the say/do ratio, agent fleet capacity, and system-level variance.
TL;DR
Agent output can climb while organizational delivery outcomes stay flat. Measurement frameworks built for human-paced pipelines miss agent fleets and fixed review capacity, so the binding constraint moves from generation to verification. This guide builds system-level measurement on DORA research, an NBER working paper on GitHub developers, and LinearB's pull-request benchmarks.
Agent PR volume and organizational delivery are measured on different instruments. Task-level benchmarks describe coding performance: Auggie, Augment Code's CLI agent, scored 51.80% on SWE-bench Pro in February 2026. Delivery outcomes have not followed. Faros AI's telemetry across 1,255 teams and more than 10,000 developers found 98% more merged pull requests alongside 91% longer review times, and no correlation between AI adoption and outcomes at the organizational level.
No fleet-level framework has become dominant as of mid-2026. Harness, in a 2026 survey of 700 developers and engineering leaders, reports that AI coding tools changed developers' daily work faster than measurement frameworks adapted, so the categories below synthesize this guide's research into a working taxonomy, not a standard one. What matters is how reliably the system turns generated code into reviewed, production-bound changes. Five categories cover that system, each answering a question about the whole rather than about a person.
| Metric Category | System Question |
|---|---|
| Engineering throughput | How much generated work becomes merged change? |
| Delivery predictability | Does planned work ship in its forecast window? |
| Say/do ratio | Does completed work match committed work? |
| Agent fleet capacity | How does generated work compare with review bandwidth? |
| Variance | How widely do outcomes spread across equivalent periods? |
Augment Cosmos, a unified cloud agents platform, is the layer these measurements describe in operational terms: agents run against shared codebase context, every action is observable and auditable, and Sessions keep the record of what each agent did.
What Are Software Factory Metrics?
Software factory metrics treat an engineering organization as a production system with measurable inputs, outputs, and constraints. The software factory is the older idea underneath them. A January 2025 US Department of Defense guidebook repeats the definition from DoD Enterprise DevSecOps Fundamentals v2.5: a software factory is a collection of people, tools, and processes that enables teams to continuously deliver value by deploying software to meet the needs of a specific community of end users. The term traces to a 1968 proposal by R.W. Bemer at General Electric and to Hitachi's Software Works, founded in 1969 and the first facility to carry the factory label.
The defining choice is system-level measurement. The SPACE framework authors call these metrics by that name because their values capture a work item's journey through the entire workflow. When one developer directs several parallel agents, per-engineer counts describe the director's activity, not the system's output. The fleet becomes the unit of analysis.
How Do You Measure Engineering Throughput When Agents Write the Code?
Engineering throughput in an agent-driven factory counts merged, production-bound changes segmented by producer, so teams don't mistake generation volume for delivered value. LinearB's 2026 benchmarks, drawn from 8.1 million pull requests across 4,800 teams, sort them three ways: agentic pull requests, which an agent generates, commits, and opens on its own; AI-assisted pull requests, which a developer authors with AI help; and unassisted pull requests. Blended reporting hides the differences.
LinearB, a vendor of engineering-metrics tooling, defines PR yield as the share of opened pull requests that merge within 30 days. A second LinearB study, covering 2.7 million pull requests at 253 organizations in early 2026, reports that among the top decile of those organizations, 79% of agent-opened pull requests merged inside the window, against 92% of human-only pull requests at the same decile.
Yield is partly a context problem. Augment Code's Context Engine semantically indexes a codebase and maps relationships across hundreds of thousands of files, so an agent opens a PR working from cross-file relationships, not the diff alone.
An NBER working paper covering more than 100,000 GitHub developers finds AI coding tools raising commits sharply, by a cumulative 40% for autocomplete, 140% for interactive agents, and 180% for autonomous agents, and those gains attenuating across the production hierarchy of lines changed, files touched, commits, pull requests, projects, and releases. The 180% commit effect falls to 50% for projects and 30% for releases. Writing code and shipping code are not the same measurement.
At the most agent-driven end of that hierarchy, OpenAI reports a five-month experiment in which three to seven engineers shipped an internal beta with zero lines of manually written code. It is a vendor self-report on one product, but it marks the boundary: even there, every production-bound change creates one validation task.
How Do You Measure Software Delivery Predictability Across Agent Fleets?
Software delivery predictability measures how reliably the system converts plans into shipped software, using commitment and forecast metrics that separate planned delivery from activity. Juan Pablo Buriticá, in a delivery metrics primer for LeadDev, is explicit: "Velocity is not a good measure of the software delivery process because it's designed to be a capacity planning tool."
Planisware recommends four numbers for a program office.
- Commitment reliability: share of planned work completed per sprint or program increment
- On-time release rate: share of releases landing inside agreed windows
- Roadmap forecast error: gap between planned and actual dates at epic or initiative level
- Cycle time and lead time: average time from commitment to production
Those four supply historical inputs for the next commitment. Probabilistic forecasting then draws on the range of historical outcomes, not a single deterministic estimate. Monte Carlo simulation produces a distribution of delivery dates, and a workable mapping aligns the 50th percentile with internal planning, the 85th with external commitments, and the 95th with contractual or regulatory deadlines. On the same logic, an item older than the 85th percentile of historical cycle time is at risk. For fleet-scale planning, LinearB's 2026 bands put planning accuracy above 82% and capacity accuracy between 85% and 115% in the top category.
What Is Say/Do Ratio in Software Engineering?
Say/do ratio is the share of committed work a team delivers, completed work divided by committed work in story points or ticket count. It is a predictability measure, not a productivity score: a fleet's PR count records output, while its say/do ratio shows whether that output fulfills a committed backlog.
The metric degrades when it becomes a target. The Scrum Alliance describes what measured teams do once predictability is scored: testing that used to live inside a story gets split out and split again, so the board fills with cards that each look smaller and safer, and people who were happy to pair or swarm start working alone because solo work is easier to attribute to a forecast. The reported result is that "the dashboards turn green. Less actually reaches the customer."
Both mechanisms transfer to agent fleets, where work is already easy to subdivide, and an agent has no swarming behavior to lose. Teams should therefore watch their own trend without ranking themselves against others, treat a ratio sitting at 100% as a hypothesis about overly safe planning, and read sustained under-delivery as a signal about the system, not a verdict on a team.
What Are Agent Fleet Capacity Metrics?
Agent fleet capacity metrics compare generated work against validated output and human review bandwidth, showing whether the system can inspect and accept what agents produce. LinearB's benchmarks cover the first half of that gap: agentic pull requests wait 5.3 times longer than unassisted ones before pickup, and AI pull requests 4.6 times longer for a first review. The second half is acceptance. Duma and colleagues, in work accepted to EASE 2026, find that most AI-generated pull requests receive no review at all, and that the reviews they do receive are dominated by AI agents, not humans.
Latency and non-review are different constraints, and throughput counts only work completed after both are clear. Where every production-bound change needs human validation, accepted throughput is capped by what reviewers process in a period, no matter how much the fleet generates. The usable measure is reviewed changes per reviewer per period, set beside agent-opened PRs per period; any surplus becomes queue. Cosmos keeps every agent action observable and auditable, which is what makes both numbers countable.
Two further measures remain proposals. Agent utilization tracks how much of the fleet's concurrency is in use, with queue depth and concurrent requests as supporting indicators. Code turnover ratio compares how often AI-generated and human-generated code is reverted or rewritten after acceptance, the one quality signal of the three. Published bands for both come from vendors and carry no dataset, so each is worth computing against a team's own history, which is what engineering capacity planning converts into staffing.
What Are Variance Metrics in Software Engineering?
Variance metrics use percentiles to quantify the spread in delivery outcomes, because the distribution, not the average, determines forecast reliability. Two teams can share a median cycle time while their upper percentiles differ, and the tail is what breaks commitments.
| Metric | Definition |
|---|---|
| Cycle time percentile spread | P50, P85, and P95 completion times |
| Lead time percentile ratio | High-percentile lead time over low-percentile, commonly P90 over P10; lower is more predictable |
| Throughput variance | Fluctuation in completed items across equivalent periods |
| Estimate variance | Gap between estimated and actual effort |
| Utilization | Share of capacity in use, the term that drives waiting time upward |
Variance belongs on a factory dashboard because queueing theory ties waiting time to it directly. Kingman's approximation for a single-server queue makes waiting time a product of three terms: variability, utilization, and mean service time, so two teams with identical service times and arrival rates can hold very different queues if one has noisier work. Waiting time then climbs sharply as utilization nears full, and a fleet run near saturation pays for variance twice. Where a team sits on that curve is what the maturity model stages describe.
How Do Software Factory Metrics Compare to DORA Metrics?
DORA measures the speed and stability of a delivery pipeline; software factory metrics measure the whole agent-plus-human system, and organizations running agent fleets need both. As of 2025, DORA's guide to software delivery metrics defines five measures: change lead time, deployment frequency, failed deployment recovery time, change fail rate, and deployment rework rate, the last being unplanned deployments that follow a production incident. The 2025 report is titled State of AI-Assisted Software Development, and it replaces the earlier elite, high, medium, and low designations with per-metric buckets and seven team archetypes.
| Dimension | DORA Metrics | Software Factory Metrics |
|---|---|---|
| What they measure | Speed and stability of the deployment pipeline | Fleet-level output and its constraints |
| Authorship assumption | A change carries human reasoning | PRs segmented by submitter type |
| Capacity model | Pipeline automation capacity | Human review bandwidth as binding constraint |
| Known blind spot | Flat outcomes despite rising PR counts | Review capacity, unless reviewers are instrumented |
The 2024 DORA report found AI adoption raising individual productivity, flow, and job satisfaction while negatively affecting delivery stability and throughput. The 2025 report found a positive relationship with throughput and with product performance, while the negative relationship to stability held. The report frames AI as "an amplifier," magnifying an organization's existing strengths and weaknesses.
A March 2026 DORA insight names the mechanism the verification tax: "Time saved writing is often re-spent auditing." The same piece reports 30% of developers having little to no trust in AI-generated code, the demand side of the review constraint. The frameworks part company on authorship: DORA assumes a change moving quickly through the pipeline carried human reasoning with it, and an agent fleet breaks that assumption without moving any pipeline number, which is where DevOps and factory models separate.
How Do You Build a Software Factory Metrics Dashboard?
A software factory dashboard combines DORA instrumentation, flow data, and agent-specific data, with fixed cadences and named owners so teams compare generated volume against accepted output. GitLab's guidance sets the precondition: standardize tagging and tracking conventions across code review, version control, deployment pipelines, incident management tools, and observability platforms before automating telemetry.
What Should You Instrument?
Instrumentation connects commit, deployment, incident, and agent-authorship data across the delivery path, so the dashboard shows where production-bound work slows or fails. Lead time for changes comes first, and it needs the full commit-to-deploy path: version-control timestamps at commit, CI build start and end times, test-stage durations, environment-promotion timestamps, and deployment confirmation. LaunchDarkly's breakdown supplies the four remaining pipeline sources.
- Deployment frequency: CI/CD pipeline logs counting successful production deployments over time
- Change failure rate: production incidents and rollbacks in a ticketing system such as Jira, against total deployments
- Recovery time: incident start and resolution timestamps from incident management tooling
- Deployment rework rate: unplanned deployments following a production incident, as a ratio against total deployments
A fifth layer sits outside DORA with no settled instrumentation: PR submitter tags, yield rate by submitter, code turnover, reviewer queue depth, and per-team agent spend. Amazon's CloudWatch Coding Agent Insights, launched in July 2026, gives an out-of-the-box view of token consumption, cost, and developer activity across an organization's coding agents. Augment Code's usage analytics track active users, lines of code by source, and agent message volume, so consumption sits beside activity.
Consumption-based measures belong in cost accounting, not performance evaluation. DORA's June 2026 insight on token-spend leaderboards makes the case: when a metric becomes a target, it stops being a good measure, and a leaderboard built on token consumption quantifies activity while ignoring outcomes.
What Cadence Should You Review At?
Review cadence should match each metric's rate of change, creating four levels instead of one standing meeting. The assignment below is a recommendation, not a published standard; expect metrics to move between levels as their real rate of change becomes clear.
- Weekly or bi-weekly: throughput, cycle time, agent yield rate, and review queue depth, which move fast enough to act on inside a sprint
- Sprint or retro: say/do ratio, forecast accuracy, and completed work reaching customers, which need a full commitment cycle to produce a number
- Monthly: developer experience survey results, which need a full month to show movement
- Quarterly: OKRs, roadmap forecast error, and satisfaction scores, matched to the planning cycle that commits against them
The schedule matters more than the specific assignment: put the reviews on the calendar before building anything, because an unscheduled metric never gets read.
Who Owns Each Metric?
Metric ownership assigns each measure to the role controlling its data and workflow, so every measure has one accountable owner. LinearB's model splits between engineering management, which owns the delivery system and the people in it, and technical leadership, which owns architectural direction and the quality of technical decisions. The agent layer is where that model breaks down, and an unassigned metric is one nobody reads.
| Metric or Responsibility | Accountable Owner |
|---|---|
| Cycle time, planning accuracy, rework rate | Engineering managers |
| Architectural direction and decision quality | Technical leads |
| Say/do ratio and forecast accuracy | Engineering managers with product owners |
| Measurement infrastructure and variance reporting | Platform teams |
| Agent-layer instrumentation and thresholds | Named per organization, no default |
Never route these metrics to individual performance reviews. Laura Tacho, an engineering leadership coach and former CloudBees engineering director, is unambiguous on measuring engineering productivity: "DORA metrics aren't intended to measure the productivity of teams, and certainly never intended to measure the productivity of individuals." The risk rises in an agent fleet, where one engineer directing several agents posts commit volume describing the tooling, not the engineer.
Start With Submitter Tags and Reviewer Queue Depth
Instrumentation order matters more than dashboard design. Tag every PR by submitter type, compute yield rate per category, and put reviewer queue depth beside agent output. Those three moves answer the question that decides whether another agent helps: is this system constrained by generation or verification? The numbers above point to verification, and it stays there until review capacity changes.
Frequently Asked Questions About Software Factory Metrics
These are the questions engineering leaders ask when instrumenting an agent-plus-human delivery system.
Related Reading
Written by
Ani Galstian
Ani writes about enterprise-scale AI coding tool evaluation, agentic development security, and the operational patterns that make AI agents reliable in production. His guides cover topics like AGENTS.md context files, spec-as-source-of-truth workflows, and how engineering teams should assess AI coding tools across dimensions like auditability and security compliance