Error Analysis for Multi-Agent Systems: What the Single-Agent Playbook Misses
The AI engineering community has, over the past two years, converged on a discipline for improving LLM products, and it deserves the adoption it is getting. The playbook: read your production traces, write open-ended notes on what went wrong, cluster those notes into a failure taxonomy, express each failure as a binary verdict rather than a quality score, and gate your releases on the failures that matter. Error analysis first, infrastructure second. It is the correct order of operations, and teams that practice it ship measurably better systems than teams that buy a dashboard and call it evaluation.
The playbook has a quiet assumption baked into it: that the trace under analysis is the work of oneagent. Every step of the loop leans on that assumption harder than it appears to. And the assumption is expiring, because the systems now being shipped are not one agent. They are a planner delegating to specialists, a coordinator fanning out to sub-agents, a researcher spawning its own workers. When the trace becomes a delegation graph, the playbook's central operation, finding the first upstream failure, stops being a reading exercise and becomes an attribution problem. This piece is about that difference, and about the failure modes that live in it.
The first-upstream-failure problem
On a single-agent trace, first-upstream-failure attribution is tractable by construction. The trace is a sequence of turns; you read it in order; the first turn where the agent does something indefensible is your answer. Downstream failures are usually cascade, so the discipline says: label the first one, fix that, re-run. The unit of analysis is the turn, and the turns arrive pre-sorted.
Now put the same failure inside a delegation graph. A planner decomposes a task and hands a subtask to agent A with an underspecified brief. A completes the brief as written, correctly, and returns a well-formed result that is subtly wrong for the actual task. A coordinator merges A's result with two siblings' work. A synthesizer drafts the final answer from the merge. The failure surfaces in the synthesizer's output, four boundaries away from its origin, and here is the uncomfortable part: every turn between origin and symptom is locally defensible. Read any single agent's transcript in isolation and that agent did its job. The trace has no first bad turn. It has a first bad handoff.
I would name the underlying mechanic error laundering: each delegation boundary re-serializes upstream mistakes into clean, confident, schema-conformant downstream inputs. A hedge in agent A's output becomes a fact in agent B's context. A missing field becomes a default. An empty result becomes "no issues found." By the time the error surfaces, it has passed through enough well-formed intermediate representations that its paper trail reads like legitimate work. Transcript order, the thing the single-agent playbook sorts by, is actively misleading here, because the symptom's position in the transcript has almost no relationship to the origin's position in the graph.
The taxonomy already exists
None of this is speculative. The academic anchor is MAST, the Multi-Agent System Failure Taxonomy out of UC Berkeley (Cemri et al., NeurIPS 2025), which did exactly what the practitioner playbook prescribes, applied to multi-agent traces: systematic annotation of real framework executions, open-ended failure notes clustered into a taxonomy, inter-annotator agreement measured before the taxonomy was trusted. The result is 14 failure modes in three categories: specification failures, inter-agent misalignment, and task verification failures. The middle category is the one with no single-agent analogue, and it is not a thin appendix; misalignment between agents (ignored input, withheld information, derailed handoffs, usurped roles) is a first-class family of ways these systems die.
What MAST established, in other words, is that the multi-agent failure distribution is not the single-agent distribution with more rows. It has its own modes, and they concentrate at the boundaries. A practitioner who open-codes multi-agent traces with single-agent categories will file boundary failures under whichever agent happened to be holding the output when the problem became visible, which is precisely the wrong place to fix.
Three failure modes from the delegation layer
Three shapes recur when we analyze delegation-layer failures. The walk-throughs below are illustrative composites: the shapes come from our authored evaluation corpora and from patterns observed in real traces, with identifying details removed. (The corpora are authored and labeled by a single judge family; our labeling-methodology note covers what that means and why we publish the limitation.)
The silent cascade
A coordinator dispatches a data-extraction subtask to a child agent. The child errors: a tool call fails, or the source is unreachable. The child returns what it has, which is nothing, wrapped politely. The coordinator proceeds as if the delegation succeeded, synthesizes the remaining children's results, and reports completion without disclosing that a branch died. The final answer is not wrong so much as quietly hollow: it is built on three legs instead of four and says nothing about the missing one.
Read the coordinator's transcript alone and it looks like competent synthesis. Read the child's alone and it looks like a routine tool failure, honestly reported. The failure exists only in the relation between them: an errored child and a parent that laundered the error into success.
The redundant delegation conflict
Two agents are handed the same subtask, sometimes by design for redundancy, sometimes because the planner lost track. They return conflicting answers: different numbers, different verdicts, different retrieved facts. The parent picks one. The transcript shows no deliberation about the conflict; the losing answer simply stops being mentioned. When the chosen answer is the wrong one, the trace contains, in plain sight, a sibling that had it right and was discarded without justification.
This mode is invisible to per-agent review by definition: both children performed well. The failure is a resolution failure, and it lives in the parent's handling of the disagreement, a place single-agent error analysis never looks because single agents do not disagree with themselves.
The synthesis failure
Every child succeeds. The parent's job is to merge their results into one artifact, and the merge is where the loss happens: one child's findings are omitted entirely, or the parent introduces a claim that appears in no child's output. The final answer cites the process ("based on the research above") while quietly departing from it. Of the three shapes this one launders best, because the output inherits the credibility of all the genuine work upstream of it.
The verdict here is only computable by comparison: final artifact against the union of child outputs. No turn-by-turn read will surface it, because no single turn is bad.
What this does to the playbook
The discipline survives; the unit of analysis does not. Three amendments, each a direct consequence of the mechanics above:
- Annotate the graph, not the transcript. Handoffs need to be first-class objects in your traces: who delegated what to whom, with what brief, and what came back. If your instrumentation records agent turns but not delegation edges, boundary failures are unrepresentable in your data before analysis even starts, and your taxonomy will silently misfile them.
- Attribute by counterfactual, not by reading order.The first-upstream-failure question on a graph is "which node, changed, un-fails the run," and answering it honestly means replaying the trace with the candidate node's output held fixed or corrected. That is more machinery than reading a transcript. It is also the only version of the answer worth acting on, because fixing the symptom node in an error-laundering chain fixes nothing.
- Give boundary failures their own binary failure modes."Every agent locally fine, system failed" is not a labeling ambiguity to be argued away; it is the signature of the delegation layer, and it deserves verdicts of its own: did the parent disclose child failures, did the resolution of conflicting children carry justification, does the synthesis contain the children. Binary, per boundary, checkable.
This is the shape of Pisama's own taxonomy: of the 87 failure modes we track, 21 are specific to multi-agent traces and 8 more to sub-agent delegation, and the delegation-layer modes above (silent cascade, redundant delegation conflict, synthesis failure) are detectors in the product, not aspirations. Those are structural counts, deliberately unaccompanied by accuracy claims: our external label corpora are mid-audit, and per-detector performance numbers are withheld until that audit completes. The methodology behind the labels, including the parts that are currently narrow, is documented in the cross-vendor judge agreement note.
The boundary is the new turn
The direction of travel in agent systems is toward deeper trees: planners over specialists over tool-runners, orchestration frameworks making delegation cheap enough to be the default rather than the exception. Every layer added moves failure probability mass from inside turns to between agents, from places where a transcript read finds it to places where only graph-aware analysis does. The single-agent playbook was the right discipline built on the right instinct: look at your data, name your failures, gate on them. The instinct carries over intact. The data model does not. For multi-agent systems, the boundary is the new turn, and error analysis that cannot see boundaries is error analysis of a system that no longer exists.