How Root Cause Analysis AI Agents Work in Production

The multi-agent RCA approach diagnoses production incidents using logs, metrics, traces, deploy history, and ownership data. Specialized LLM agents isolate each signal type and test hypotheses, while a supervisor correlates evidence across modalities and keeps observability signals in bounded contexts. During correlation, the supervisor checks hallucinated dependencies and missed evidence, and each agent stays scoped to its assigned signal.

Why On-Call Engineers Need More Than a Single-Agent RCA Bot

Single-agent RCA bots lose diagnostic reliability under production incident pressure. Logs, metrics dashboards, log aggregators, trace viewers, deploy history, and service ownership data accumulate in one reasoning path during 2 a.m. diagnosis. Root cause analysis is “a critical part of the incident management process” and “a demanding task for on-call engineers,” according to a study by Roy et al. on cloud incident RCA. Traditional automated RCA “has relied on human engineers to manually correlate these fragmented signals,” which remains labor-intensive and error-prone, according to a ScienceDirect survey on cloud RCA.

Single-agent LLM approaches lose iteration and refinement when one LLM handles planning and execution. In that pattern, “the LLM generates a single unstructured text response attempting to address all objectives. No iteration or refinement occurs,” according to research on structured incident brief generation. Multi-agent RCA systems assign observability and validation work to scoped agents, then coordinate the outputs before a final diagnosis. This guide covers architecture, orchestration, integrations, accuracy numbers, and production limitations.

What a Multi-Agent RCA System Is and How It Differs From Single-Agent RCA

A multi-agent RCA system deploys LLM-driven agents with distinct roles or data scopes. A supervisor sequences execution and correlates findings. LATS-RCA “consists of two diagnostic agents (log and metrics), coordinated by a supervisor,” with the log agent exploring logs and the metric agent analyzing time series and visualizations. OpsAgent and similar systems consume metrics, logs, and distributed traces.

Four structural differences separate multi-agent RCA from single-agent RCA architectures:

• Modality specialization: The log agent uses “custom tools for file listing, content retrieval, and pattern-based search”, while the metric agent uses tools for time-series loading, comparison, and visualization.
• Parallel hypothesis maintenance: LATS-RCA “explicitly maintains multiple candidate root causes during search and incrementally gathers evidence from logs and metrics.”
• Context isolation: Each sub-agent operates in its own isolated context, a pattern discussed in more depth in the single-agent vs multi-agent AI guide.
• Cross-modal correlation: The supervisor performs final correlation after scoped diagnostic work.

These structural differences make multi-agent RCA useful when production incidents require scoped evidence gathering before final correlation.

Older neural and similarity-matching RCA methods “rely on surface-level similarity matching or prompt engineering” and “are not able to dynamically collect additional diagnostic information such as incident related logs, metrics or databases.” Multi-agent LLM systems can iteratively collect incident-related logs, metrics, and databases when the incident platform exposes those sources to the agents. The mechanism is tool-based retrieval plus supervised correlation, and the boundary is the connected telemetry, database, and ownership data available during the incident.

Named Multi-Agent RCA Systems

Named multi-agent RCA systems separate log, metric, validation, reporting, and remediation roles. Incident teams can compare architecture patterns against stated benchmark boundaries. The research field includes academic systems and vendor products. The table marks vendor claims as vendor claims.

Sources: LATS-RCA, OpsAgent, Flow-of-Action, MA-RCA, and Datadog Bits AI SRE. On OpenRCA, Claude 3.5 Sonnet achieved 11.28% accuracy with the RCA-Agent method, which OpsAgent v3 describes as evidence of IM’s inherent difficulty and current approach limitations.

How Agent Orchestration and Task Decomposition Work

Orchestration in multi-agent RCA assigns diagnostic subtasks to specialized agents and coordinates outputs through a supervisor or planner. This matters during incidents that span multiple telemetry sources because teams need to compare evidence before a final diagnosis. Research on AgentTrace defines a 3-5 agent structure with “a Coordinator (task decomposition and delegation), Specialists (domain-specific tasks), a Reviewer (output validation), and an Executor (final actions).”

A multi-agent RCA run follows a handoff sequence before the final incident brief:

1. Decompose the alert into log, metric, trace, change-history, and ownership questions.
2. Delegate each question to the specialist agent with the relevant tool boundary.
3. Review specialist outputs through a reviewer, verifier, or supervisor step.
4. Correlate final evidence into a root-cause hypothesis, confidence score, and remediation boundary.

This orchestration flow keeps agent work inspectable before any diagnosis reaches an on-call engineer.

The Supervisor Pattern Coordinates Production RCA Systems

The supervisor pattern coordinates production RCA agents by centralizing quality control while subagents remain stateless. Incident teams get one point for handoff, sequencing, and final correlation. This supervised decomposition appears in research systems, cloud architecture guidance, and vendor implementations. LATS-RCA uses a supervisor for cross-modal handoff, execution sequencing, and final correlation. Microsoft Azure’s AI agent design patterns document a manager that “creates an initial diagnostic plan, consults specialized sub-agents, and adapts the plan in real time.” AWS ships “four specialized AI agents working together under a supervisor agent.” The tradeoff is latency: a lightweight SupervisorAgent reports an “average latency increase of 37.27%.”

The table below compares supervised decomposition against related coordination patterns, showing where each design places validation and what it trades for that placement.

Sequential and Role-Based Decomposition

Role-based RCA decomposition assigns diagnosis, remediation planning, and risk assessment to separate agents. This separation lets incident briefs preserve bounded reasoning across production workflows. Some systems decompose incident analysis into single-objective calls: a Diagnosis Specialist, Remediation Planner, and Risk Assessor, with “a coordinator (non-LLM orchestration logic)” producing a structured incident brief. MA-RCA assigns retrieval, analysis, validation, and reporting agents. Its Retrieval Agent “proactively queries a historical case repository” and reaches external benchmark results of Nezha Acc=0.958, F1=0.952 and Power Acc=0.843, F1=0.828.

Comparing role-based RCA against other design points shows how trigger conditions, primary inputs, and operational limits shift across incident types.

Modality-Specific Agent Design and the Log Volume Problem

Modality-specific RCA agents filter high-volume log data before metric and trace correlation. Specialists can then compare observability signals without overloading a single reasoning path. KnowledgeMind describes a Log Agent that performs preliminary filtering because logs consume more processing capacity than metrics and other observability data, using Drain, keyword extraction, clustering, and knowledge base matching. The LATS-RCA paper also notes that logs appear in JSON, unstructured text, and mixed formats, while metrics come from diverse Prometheus exporters with service-specific naming.

Progressive Discovery Over Pre-Scripted Paths

Progressive discovery RCA follows telemetry anomalies, deployment history, and code diffs step by step when those data sources are available. Investigations adapt to incident evidence. Microsoft describes an Azure SRE Agent that runs progressive discovery, where “each step, from metric anomaly to deployment history to a specific diff, followed from what the previous step revealed. It was not a pre-scripted path.” Similarly, incident.io’s multi-agent investigation system runs “parallel searches, generates findings, formulates hypotheses, asks clarifying questions through sub-agents, and presents reports in Slack within 1-2 minutes.”

[ Free report ]

The Agentic SDLC

How teams like Stripe, Ramp, and Uber move from solo coding agents to a coordinated, team-level system.

Download the guide

What Data Sources and Tool Integrations Production RCA Agents Use

Multi-agent RCA systems consume metrics, traces, logs, events, topology graphs, and change history, often through the Model Context Protocol (MCP). OpenRCA covers CPU usage, response time, APM error rate, latency, traces, and component logs. Events include restarts and deployments, and topology maps blast radius. Resolve.ai indexes “every deploy, configuration edit, feature flag transition, secret rotation, and data migration with timestamps and ownership.” The OpenRCA Controller analyzes metrics, traces, then logs after anomaly detection, fault identification, and localization.

Topology Supplies Dependency and Blast-Radius Context for Multi-Service RCA

Topology is a distinct RCA input for multi-service incidents. Logs and metrics lack dependency ownership, customer impact, and change-history boundaries needed for blast-radius analysis. As Anyshift argues in a piece on AI SRE and topology, “an LLM with only logs and metrics can’t answer key questions: What else depends on this? What’s the blast radius? Which customers are affected?” A structured topology graph, described as “a continuously updated, queryable graph of infrastructure and application dependencies, enriched with ownership, change history, and business metadata,” supplies the data needed to answer them. A service dependency graph from traces differs from a causal graph, which captures “how load or latency changes propagate,” as covered in more depth in the AI agent incident response guide.

MCP as a Listed Access Pattern

MCP-based RCA integrations connect agents to observability tools through standardized queries. Diagnostics can retrieve logs, metrics, traces, and service state from the platform APIs listed below. These platforms document MCP capabilities for RCA agents querying observability systems.

The Auggie CLI ships with MCP support and OAuth-backed servers for GitHub, GitLab, Jira, Linear, Confluence, Notion, Slack, Vercel, Cloudflare, Stripe, Sentry, and 100+ third-party services, so RCA agents can pull code, tickets, and infrastructure state through the same standardized tool call surface.

Sources: Honeycomb MCP docs, Datadog LLM Observability MCP server, New Relic AI MCP server launch, groundcover on AI agent observability, and the Dynatrace MCP observability hub listing.

Runtime Telemetry Alone Falls Short

Telemetry-plus-code RCA combines runtime signals with repository evidence for code-change incidents. Agents can move from symptom correlation to localizing failures in specific commits, functions, or configuration changes. Code knowledge measurably improves diagnosis. ICSE 2025 research demonstrates that “incorporating code knowledge improves root cause localization by 28.3% over the prior leading method.” A DevOps function-calling example from Muhammad Raza shows an agent calling get_workflow_logs(), then analyze_recent_commits(), then search_similar_issues() to synthesize a root cause report.

Telemetry-plus-code RCA also benefits from repository-wide context retrieval. The Context Engine grounds investigations across 400,000+ file codebases, retrieving commit history, docs, tickets, design decisions, previous discussions, and related issues so agents can see the code and the conversation around it in the same session.

How Code and Repository Context Improve RCA Accuracy

Code and deploy context improve RCA accuracy by letting agents correlate incident onset with specific changes. Temporal correlation shifts a finding from “latency is high” to “latency spiked shortly after a deploy that modified database connection pool settings”. Change intelligence indexing proposes ranked changes in the relevant time window when symptoms begin.

The Praxis paper demonstrates the impact quantitatively through LLM-driven traversal over a Service Dependency Graph and a Hammock-block Program Dependency Graph. The results below compare an SRE-Agent baseline against graph-augmented configurations, holding models constant within each pairing so the graph traversal effect is separable from model choice.

Specialized code-context agents add graph navigation and verifier roles to the code-context RCA pattern. GAMAR enriches dependency graph nodes with health insights and deploys a Navigator Agent, Diagnoser Agent, and Verifier Agent, reaching an average F1-score of 88.4%, “outperforming state-of-the-art baselines by 4.6%.” AgentFL applies a related validation pattern: Comprehension, Navigation, and Confirmation agents identify, narrow, and validate faults with debugging tools.

Teams that rely solely on LLM reasoning for code-change RCA encounter accuracy ceilings when dependency propagation across services and code paths is absent. The Context Engine processes entire codebases across 400,000+ files through semantic dependency graph analysis, and RCA agents can use that graph context for code-change attribution.

Verified Accuracy Results From Academic and Industry Research

Multi-agent and LLM-based RCA systems report external accuracy results from single digits on hard benchmarks to above 90% on curated datasets. Reasoning-quality problems persist even when final answers are correct. The numbers below retain source context so engineering teams can weight claims appropriately.

• The RCACopilot paper (Microsoft, EuroSys ’24) reports up to Micro-F1 = 0.766 on one year of Microsoft incidents, deployment in 30+ teams, and evaluation limited to the Transport team’s dataset.
• LATS-RCA reports 91.3% accuracy on LO2 across 37 real-world incidents over six months, with each incident spanning 53,000-217,000 log lines and 300-800 metric time series.
• SynergyRCA achieves “average precision of 0.88 and 0.92” on two production Kubernetes datasets and pinpoints issues in approximately 2 minutes.
• The ReAct Agent for RCA (Roy et al., FSE 2024) reports lower zero-shot, out-of-domain results: CoT accuracy 39%, Retrieval-Based (k=10) 39%, ReAct S+Q BM25 35%.

Engineering teams should compare these accuracy results by benchmark scope, telemetry volume, and whether the system validates recovery actions after diagnosis.

The Diagnosis-to-Recovery Gap Appears in Recovery Validity Results

The diagnosis-to-recovery gap appears in R2Act’s 302 quality-audited Kubernetes incidents. High root-cause service accuracy does not reliably produce valid remediation actions. R2Act found that “the strongest RAG-based LLMs reach 91.4%-99.7% root-cause service accuracy, yet their recovery validity remains only 36.8%-60.3%.” Even when diagnosis is correct, “recovery-oriented methods still choose invalid actions for 39.5%-62.0% of correctly diagnosed incidents.”

Reasoning Misalignment Persists Even in Correct Answers

Correct outputs often come from flawed reasoning. Qwen-flash, Qwen-turbo, and SRE-Qwen-32B achieve below 42% RCA accuracy with reasoning misalignment rates of 76.8%, 65.8%, and 69.2% respectively. The End-to-End RCA benchmark concludes that “even correct answers often arise from misaligned reasoning.” Benchmark scope varies across studies. RCAEval aggregates 735 cases, OpenRCA covers 335 real failure cases with 68GB of telemetry, and R2Act covers 302 Kubernetes incidents.

How Multi-Agent RCA Systems Handle Hallucination and Trust

Multi-agent RCA systems target hallucination risk through specialized decomposition, adversarial verification agents, evidence grounding, and confidence scoring that routes low-confidence outputs to human review. These controls reduce risk without eliminating it, so teams should gate automation by confidence and human review.

Trust controls in multi-agent RCA usually appear as four checkpoints across the diagnostic path:

1. Scope restriction narrows each agent to a telemetry modality, code context, or validation role.
2. Stored evidence keeps investigation facts outside the LLM’s transient reasoning state.
3. Adversarial verification tests findings with counterfactual review before reports reach engineers.
4. Confidence routing sends low-confidence findings to human review instead of autonomous action.

These controls reduce blind trust in one generated answer while still requiring measured rollout.

Specialized decomposition narrows each agent’s domain. MA-RCA reports that “each agent operates within a dedicated domain, minimizing context switching and reducing hallucination risks.” Razorpay’s Project Viveka grounds analysis in stored facts by placing investigation evidence in Memory and having the Supervisor reason over concrete evidence rather than relying on the LLM to retain the full investigation state.

Model routing is another lever. Prism, the routing layer inside Cosmos, sends diagnostic work across Claude, GPT, Gemini, and open-source models so each step does not depend on one model path. Combined with the Context Engine, this routing contributes to a 40% reduction in hallucinations for coding and diagnostic work.

Verification and Adversarial Validation Agents

Dedicated verifier agents catch hallucinations before reports reach engineers. GAMAR’s Verifier Agent “enforces an Adversarial Validation Protocol to mitigate hallucinations through rigorous counterfactual reasoning” and achieves an average F1-score of 88.4%. Slack’s security investigation system uses a Critic as a “meta-expert” whose “responsibility is to assess and quantify the quality of findings” and assign a credibility score.

Evidence Grounding and Confidence Scoring

Grounding techniques force answers to cite specific evidence. incident.io’s RAG approach produces “a response with citations, pointing back to specific log lines or commit hashes it used to reach its conclusion.” OpenRCA uses code execution: the LLM generates and executes Python code to analyze telemetry data programmatically, so raw telemetry does not have to remain in prompt state. Techstrong’s production deployment guide recommends structured JSON including a “confidence”: 0.82 field, with routes to human review if conditions fail.

Progressive Autonomy Is the Consistent Recommendation

Progressive autonomy limits multi-agent RCA risk by moving agents from incident summaries to read-only diagnostics to low-risk automation. Teams build trust through measured success before autonomous remediation. CSO Online describes the path as “progressive autonomy. Let the agent start by summarizing incidents, correlating changes and suggesting next steps. Then move to read-only diagnostics. Only after consistent success should it be allowed to trigger low-risk automation.” A staged autonomy model runs from Stage 1 read-only anomaly detection to Stage 4 autonomous action within guardrails, as covered in the AI SRE guide on progressive autonomy, where each stage requires “trust history built through measurement.”

Key Limitations and Failure Modes That Persist Across Models

Multi-agent RCA failures are often structural and come from the agent framework rather than the underlying model. Analysis of 1,675 agent runs across five LLM models on OpenRCA found a framework-level pattern: “Hallucination in Interpretation (71.2%) and Incomplete Exploration (63.9%) persist across all tested models at comparable rates regardless of capability tier.”

Several failure categories recur across the research:

• Cascading errors between agents: MetaGPT’s pipeline “collapsed to R_Sf as low as 0.0%, as a single corrupted output propagates downstream unchecked.”
• Specification and coordination breakdowns: UC Berkeley’s MAST taxonomy attributes 41.8% of failures to Specification Issues and 36.9% to Inter-Agent Misalignment. The MAST paper also found that “using the same model in a single-agent setup often outperforms multi-agent versions.”
• Cost multiplication: A Galileo analysis on multi-agent failures shows “a task that costs $0.10 in API calls for a single agent might cost $1.50 for a multi-agent system,” driven by repeated state sharing across agents.
• Attribution errors: A major retailer’s two-year postmortem study found “a persistent ~10% attribution error rate: the model blamed technologies simply because they were mentioned in the incident thread.”

These recurring failures make supervision, evidence retention, and bounded deployment more important than model upgrades alone.

The Toil Contradiction

The toil contradiction occurs when AI RCA adds supervision work while automation claims promise MTTR reduction. SRE teams need survey-level boundary conditions before treating vendor metrics as toil reduction. The Catchpoint SRE Report 2025 found median SRE time on operational tasks rose from 25% to 30%, the first increase in five years, and toil increased for 43% of organizations “despite widespread tool adoption.” The 2025 DORA Report found burnout “unaffected by AI.” AI SRE adoption creates its own supervision overhead, as covered in the AI SRE toil analysis: “supervising agent output quality, maintaining models, running the infrastructure underneath the agents, and reviewing decisions that previously did not need reviewing.”

How Multi-Agent RCA Fits Into Incident Management Workflows

Multi-agent RCA integrates across alerting, on-call routing, ChatOps investigation, runbook execution, and postmortem drafting. The agent triggers at the alert layer, then surfaces findings where engineers already work.

At detection, Logz.io’s AI Agent provides “immediate contextual awareness on anything for which you’ve created an alert” and distinguishes cause from symptom. At routing, PagerDuty’s Fall ’25 AI Agent Suite includes an SRE Agent that “learns from related incidents, automatically surfaces context, recommends and executes diagnostics and remediations” and “generates self-updating runbooks.” Together, these examples show how RCA agents move from alert explanation into routing and diagnostic execution.

ChatOps is where investigation and remediation converge. incident.io’s AI SRE in Slack “analyzes recent PRs merged to production, correlates timestamps with alert onset, and surfaces the likely deployment,” and “can even generate a fix and open a pull request directly from Slack.”

For read-only exploration, the Auggie CLI ships an ask mode (--ask) for retrieval and non-editing tools and a separate Quick Ask Mode toggle in local agent mode. This read-only boundary fits early RCA workflows, and parallel tool calls make tool-call turns at least 2x faster while approval gates control terminal commands.

Augment Cosmos gives incident teams the primitives to run these workflows as governed, auditable pipelines rather than one-off prompts. Environments define where agents run and what they can touch. Experts define how agents behave and what events they subscribe to. Sessions turn one-off prompts into replayable workflows with Human-in-the-Loop policies, cutting the SDLC from eight human interruptions to three checkpoints while keeping approval gates in place for production actions.

The table below contrasts an ad-hoc ChatGPT-style workflow with a purpose-built AI SRE integration, showing where data access, hallucination checks, and remediation boundaries diverge in practice.

Source: incident.io on AI root cause analysis testing.

For postmortems, incident.io auto-drafts from the captured timeline, “requiring about 15 minutes of editing rather than 90 minutes of reconstruction.” Its ~80% automation applies to procedural and documentation work rather than technical troubleshooting. AI RCA accuracy is also “substantially higher for incidents matching known historical patterns than for novel failure modes,” as covered in the AI SRE incident management guide.

Ground Your RCA Agents in Code Before Trusting Them in Production

The diagnosis-to-recovery gap and framework-level hallucination rates point to one conclusion: multi-agent RCA is only as reliable as the context its agents reason over. Telemetry identifies symptoms, and code and dependency context link those symptoms to code, configuration, or deployment changes.

Start by mapping your dependency graph before wiring agents into production incident flows. The Context Engine processes entire codebases across 400,000+ files through semantic dependency graph analysis and delivers 5-10x speed-up on multi-file investigation tasks, giving RCA agents the context they need to connect symptoms with code and deployment changes. Cosmos then wraps that context in Environments, Experts, and Sessions so those investigations run as governed, auditable workflows across the software development lifecycle.

FAQ

Related Reading

• 5 Best AI SRE Tools in 2026: A Practitioner's Shortlist
• Best Incident Management Software (2026)
• 8 Best AI Coding Assistants (Updated May 2026)
• 11 Best AI Coding Tools for Enterprise
• 7 SOC 2-Ready AI Coding Tools for Enterprise Security