Failure detection for production AI agents

Your AI agents fail silently.
Pisama catches it and shows you the fix.

Agents return 200 and still get it wrong: loops that never break, state quietly corrupting, one bad output cascading downstream. Pisama reads every production trace, names the exact agent and step that failed, and hands you the fix. Works with the frameworks you already run.

Failure mode 1 of 5illustrative
Infinite loop

An agent repeats the same state transitions. Every call returns 200.

A research agent calls search, gets insufficient results, rephrases, gets similar results, rephrases again. No exception, no timeout, and your dashboard stays green.

caught at T1 · hash & subsequence matching · no LLM
San Francisco·MIT core · TypeScript + Python·25 production-grade detectors
§ 03 · What it does

Catch it. Explain it. Fix it.

The SDK runs on your machine and finds the failure. The platform writes the patch back into your repo. Both ship in the open — see exactly which parts cost money and which don't.

I

It catches what you missed

SDK · local · $0

Loops, hallucinated tool calls, persona drift, runaway costs, corrupted state. Detectors run on every trace, 25 externally validated at production grade. The ones tuned for structure cost nothing and never leave your machine.

II

It explains in plain English

SDK · per-issue

Each failure comes with what broke, where it broke (the exact agent and step), and a suggested fix. No stack-trace archaeology, no eyeballing a 4k-token transcript.

III

It writes the fix back into your code

Platform · LangGraph live

For LangGraph recursion limits, Pisama opens the fix as a GitHub PR you can review and merge — shipping today. Auto-fix for n8n, Dify, OpenClaw, and Anthropic Managed Agents on the roadmap. Hosted at pisama.ai.

Detect & Diagnose ship in the open-source SDK. Heal is live on pisama.ai for LangGraph; more runtimes shipping.
§ Two pressures, one runtime

Operations and accountability.

Pisama runs while you’re not watching. That matters to the team shipping agents and to the people who answer for them.

Platform teams

You can’t watch every agent run in production.

  • Silent

    Agent says it finished. Output is wrong. You find out from a user.

  • Token burn

    Loops uncontrolled while you sleep. The bill arrives later.

  • Trust break

    After one bad incident you start checking every turn. Automation defeated.

Enterprise

Your agents have no accountability layer.

  • No audit

    Legal asks what the agent did. The trail lives in four tools, not one.

  • Scope creep

    Agent acquires more agency than it was shipped with. Nobody notices.

  • Cascade

    One bad output poisons every downstream agent. No circuit breaker.

§ 04 · Verticals

Detection tuned to the workflow, not just the framework.

Five narrow, domain-specific calibration packs in active R&D. Discovery stage today: no calibrated threshold, no customer traces adjudicated, published honestly as exactly that.

Discovery stage

Accounting & Tax

Reconciliation and Posting Integrity

A bank-reconciliation workflow: human approval, authoritative ERP posting outcomes, and completion-exception state, checked against records the agent cannot edit.

Discovery stage

Cybersecurity

Advisory Citation Integrity

Whether an agent correctly decided a published security advisory applies to a deployed asset: version-range logic, CVSS scoring, and advisory revision lineage.

Discovery stage

Financial Risk & Compliance

SEC Filing Fact Integrity

Whether an agent's citation of a reported financial fact matches the filing itself: scale, sign, rounding, and restatement lineage against SEC EDGAR source data.

Discovery stage

Insurance

Commercial Submission Integrity

A Commercial Auto submission workflow end to end: public source-fact lineage, broker review, and carrier transport receipts.

Discovery stage

Legal

Research and Citation Integrity

Whether an agent's legal-research citations bind to the real source: the exact opinion, version, pinpoint, and quoted text.

§ 05 · Compatibility

Reads what you already write.

Drop-in adapters for the frameworks, runtimes, and editors you already use, plus an MCP server and generic OpenTelemetry ingestion for everything else.

Cursor / Claude Desktop / WindsurfClaude CodeLovablev0BoltReplit AgentLangGraphClaude Agent SDKn8nDifyOpenClawClaude Managed Agents
+ any framework emitting OpenTelemetry: OpenAI Assistants, AWS Bedrock, Google ADK, LangChain Deep Agents, CrewAI, AutoGen, and more.
§ 06 · The catalogue

Detectors for single-agent, multi-agent, and sub-agent systems. Six categories.

The named detectors, grouped by category. Each is a calibrated pattern-match against a specific failure shape, not a generic rubric. Plus framework-specific packs that know what goes wrong inside the runtimes themselves.

Planning & Decomposition

6
decompositionspecificationdelegationworkflowroutingdispatch_async

Execution & State

7
loopcorruptionoverflowpropagationmemory_stalenessparallel_consistencycompletion

Coordination

6
coordinationcommunicationmulti_chainsubagent_boundaryorchestration_qualitytask_starvation

Verification & Quality

7
hallucinationgroundingcontextcitationentity_confusionretrieval_qualitycritic_quality

Behavior & Safety

7
persona_driftderailmentwithholdinginjectionapproval_bypasscowork_safetyexploration_safety

Reasoning & Observability

5
convergencereasoning_consistencyadaptive_thinkingcompaction_qualitymodel_selection
Framework-specific packs+5LangGraph+5OpenClaw+3n8n+2Dify= 15 framework-specific detectors
25 externally validated at production grade
§ 07 · The method

Five tiers. Heuristics first. LLMs and humans only when forced.

Fast detectors handle 90%+ of detections at zero cost. The pipeline escalates only when a tier can't conclude.

T1

Hash

Identity matching on transition graphs. Loops, deadlocks, repetition.

p50
~0 ms
cost
$0
T2

Delta

Type, null, oscillation tracking. Element coverage on cross-agent payloads.

p50
~1 ms
cost
$0
T3

Embeddings

Behavioral embedding of outputs vs. embedding of the role.

p50
~10 ms
cost
$0
T4

LLM judge

Escalation tier. Invoked only when T1–T3 disagree or are ambiguous.

p50
~200 ms
cost
~$0.02
T5

Human

Async review for edge cases. Optional, opt-in.

p50
async
cost
90%+ of detections resolve in T1–T3 at $0. T4 uses your own ANTHROPIC_API_KEY when invoked. T5 is a human review queue.
§ 08 · Defense in depth

Layered by cost. The cheap checks run on every step.

Running the whole agent and reading the transcript afterward pays full price just to learn what broke. Pisama checks while the run happens, cheapest tier first, and pays for a model only where the free tiers cannot decide.

L0PreventionGuardrails distilledfrom past verified fixes$0L1In-flight checksLoops, drift, corruption.Every step, as it runs.$0L2LLM judgeGray zones only, whenthe tiers disagree.~$0.02L3AttributionWhich agent, whichstep, and why.includedL4HumanAsync review foredge cases. Opt-in.queueHEAL · VERIFIED FIX BECOMES A GUARDRAIL
● Amber layers run on every step at $0.Muted layers are invoked only when the cheap tiers cannot decide.
Live vs roadmapPrevention guardrails, in-flight checks on every step, forensic attribution, the LangGraph pull-request writeback, and the n8n replay-and-apply loop are live today. Pisama flags the failure in-flight and drives the fix. Acting on that signal automatically, steering a run back on its own while it is still going, is on the roadmap.
Why not something simpler

A final-output eval

Grades the finished output, pass or fail. You have already paid for the whole run, and the score cannot tell you which step broke or halt a loop while it spins.

Observability

Records every span faithfully, then decides nothing. You still have to know what to look for and rule the run good or bad yourself.

A retry

Runs it again with the same prompt and the same tools, so it meets the same failure. A retry with no diagnosis just pays twice.

The cheap tiers are the whole point. Hashing transitions, tracking deltas, and embedding outputs cost nothing, so Pisama can check every step of every run. A judge that read every step could not: the latency and the bill would make it unaffordable. Pisama calls a model only in the gray zones the structural tiers flag.
Detector cost telemetry

The current optional metric compares aggregate process-local LLM judge spend with validation events recorded through one endpoint. Those streams are not linked by detection or cohort. It does not measure task or business outcomes, omits other review paths, and resets with the process. A durable cost-to-outcome view still requires an independent task verifier and an append-only usage ledger.

§ 09 · Public packages

Twelve public packages. Python and TypeScript. Use what fits your stack.

pisama

The Python entry point: offline analysis, CI checks, trace scrubbing, CLI, and local MCP server.

pip install pisama · MIT
pisama-core

Detection orchestrator, trace models, scoring, and registered heuristic detectors.

pip install pisama-core · MIT
pisama-detectors

A standalone detector library with framework packs for LangGraph, Dify, n8n, and OpenClaw.

pip install pisama-detectors · BSL 1.1
pisama-auto

Zero-code auto-instrumentation for popular Python agent frameworks.

pip install "pisama[auto]" · MIT
pisama-agent-sdk

ATIF trace adapters and real-time failure hooks for agent runtimes.

pip install pisama-agent-sdk · MIT
pisama-claude-code

Local Claude Code trace capture, token and cost tracking, exports, and opt-in sync.

pip install pisama-claude-code · MIT
pisama-verifier-gym

Audit harnesses, datasheets, agreement metrics, and regression gates for verifiers.

pip install pisama-verifier-gym · MIT
pisama-n8n-engine

The dependency-free structural detection engine behind Pisama for n8n.

source package · Fair-code
@pisama/sdk

TypeScript middleware, OpenTelemetry export, and PII redaction for AI SDK applications.

npm i @pisama/sdk · MIT
@pisama/detectors

Composable TypeScript failure detectors used by the SDK and dashboard.

npm i @pisama/detectors · MIT
@pisama/cli

Project setup, verification, MCP, and Harbor ATIF analysis for TypeScript teams.

npx -y -p @pisama/cli pisama-ts init · MIT
n8n-nodes-pisama

The n8n community node for signed execution delivery to hosted or self-hosted Pisama.

install in n8n · MIT
§ 10 · Common questions

Common questions.

Q.1

How is this different from observability and trace tools like Langfuse or LangSmith?

Trace stores record what your agent did and hand you the spans. You still have to know what to look for and decide something went wrong. Pisama makes that call: point it at the same traces and it names the step that failed and why, with a calibrated confidence, then gives you the fix. Tracing is the input; Pisama is the detection-and-fix layer on top. Keep your trace store and point Pisama at it.

Q.2

How is this different from rubric-based LLM judges in Bedrock / Foundry / Vertex?

Those judge the artifact: was the output good? Pisama detects what happened during execution: loops, state corruption, persona drift, coordination breakdown. Different layer; complementary tools.

Q.3

Is Pisama an eval tool?

No. Evals grade output quality, and they sit a layer above Pisama. Pisama detects structural process failures in running agents and reports them as binary failure modes with calibrated confidence, not generic quality scores. If you run evals, keep them; Pisama watches the layer your rubrics cannot see.

Q.4

Why not just grade the final output, or retry the failing run?

Because both act too late. A final-output eval pays for the whole run first, then hands you a pass or fail with no where and no why, and it cannot stop a loop while it is still spinning. Observability records every span but decides nothing, so you are still the one reading the transcript. A retry re-runs the same prompt with the same tools and meets the same failure, paying twice for one bug. Pisama sits earlier and cheaper: the structural tiers cost $0, so it checks every step as the run happens, names the step that broke, and calls a model only in the gray zones a cheap check cannot settle. Grade the output too if you like; Pisama watches the process the grade cannot see.

Q.5

Does Pisama send my traces anywhere?

The T4 LLM judge is opt-in and uses your own API key. Pisama does not proxy your model traffic, and PII redaction runs before traces are stored.

Q.6

What if I'm not on a supported framework?

If your traces have transitions, shared state, and message history, the detection methods apply. We ship dedicated adapters for 12 frameworks/runtimes/editors, plus generic OpenTelemetry ingestion; anything that emits OTel (CrewAI, AutoGen, others) works out of the box.

Q.7

Why heuristics over an LLM judge?

Structural process failures (loops, state corruption, coordination breakdown) have a fixed shape an LLM judge has to read and reason about every time. A heuristic detector finds the same shape in microseconds, for $0, and never truncates a long trace. Genuinely ambiguous cases still escalate to an LLM judge; heuristics handle the majority of detections first.

Q.8

What does Pisama miss?

Genuinely ambiguous cases, where even careful human labellers disagree, are surfaced as advisory, not as flagged failures. We do not claim to catch everything.

§ Verdict · Platform

Stop finding out
from your users.

Pisama catches the failures that still return 200: loops, silent corruption, scope creep, cascades. Across every framework you orchestrate.

§ 11 · Verdict · Enterprise

Stop explaining to legal
what your agent did.

Signed audit trail, scope containment, regulator-grade retention. The accountability layer your customers will eventually require.

See the enterprise pitch