pisamaBack to Pisama

Know what can go wrong.

105 detectors, organized by the failures they catch. Find a category, explore an example, and see the status of every detector.

Detector coverage · Updated September 21, 2026

64Production10Beta11Experimental20Below release gate

Showing 20 of 105 detectors

Context, memory, and state1

Used out of date facts

Below release gate

Catches an agent that retrieves old material and uses it as though it were current.

For exampleAn agent follows an old shipping policy even though its memory also contains the newer replacement.

memory_staleness

Coordination and delegation2

Approved obviously broken work

Below release gate

Catches a reviewer agent in a reflection loop that approves work it should have sent back.

For exampleA reviewer approves a payment integration even though the charge function is still an empty placeholder.

critic_quality

Delegated without any instructions

Below release gate

Catches handoffs that were underspecified at the moment they were made: no definition of done, no bounds on time, cost, scope, or authority, and instructions vague enough that the receiver has to guess.

For exampleTold to 'sort out' a billing issue with no account, timeframe, or limits, the worker issued its own refund.

delegationEvaluation data under review

Dify1

Sent a request to the wrong team

Below release gate

Catches a Dify question_classifier node that routes a request badly.

For exampleA Dify workflow routes a refund request to technical support instead of billing.

dify_classifier_drift

Grounding and retrieval3

Buried the answer in noise

Below release gate

Measures what fraction of the retrieved context was actually relevant to the query, and fires when most of it was not.

For exampleAnswered a pricing question after retrieving eight pages, only two of which were actually about pricing.

context_precision

Made claims no source backed

Below release gate

Verifies an answer against the source documents it was supposed to be built from, with the emphasis on numbers, named entities, and quoted attributions.

For exampleReported the wrong revenue figure and credited the forecast to an analyst who never said it.

grounding

Pulled up unrelated passages

Below release gate

Scores every retrieved chunk against the query that fetched it and reports the ones that do not address the question.

For exampleAnswered a policy question mostly from unrelated pages, with only one thin excerpt actually on topic.

chunk_relevance

Managed agents1

Stopped making progress

Below release gate

Catches a session that has stopped making progress: a long silence between consecutive events, a session sitting in running or rescheduled without a single progress event, or a long total duration with almost nothing done in it.

For exampleA session remains marked as running but produces no progress events for a long period.

ma_session_stall

OpenClaw1

Created too many child agents

Below release gate

Watches the session.spawn events in one session's own event stream for delegation that runs away from its parent.

For exampleOne session spawned five child sessions, past the safe limit, including one that pointed back at itself.

openclaw_spawn_chain

Progress and looping2

Kept running long after breaking

Below release gate

Reads a recurring job's run history instead of one run, and flags degradation across the sequence: output that stops changing, latency creeping up, executions being skipped, and errors that climb or swallow the whole history.

For exampleA report intended for every weekday is scheduled for once a month.

scheduled_task

Retried without getting closer

Below release gate

Watches a numeric metric an agent is trying to improve across iterations and flags when that series stops improving.

For exampleKept tuning for twelve more rounds after results stopped improving, burning the rest of the budget.

convergence

Safety and permissions1

Used tools it wasn't given

Below release gate

A subagent uses a tool outside the set its parent authorized, tries to spawn a child of its own, or drifts far enough from the parent's instruction that its output no longer relates to it.

For exampleA child agent asked only to inspect a file proceeds to modify and publish it.

subagent_boundary

Task fidelity3

Overthought it, or barely tried

Below release gate

Flags runs where the model's reasoning spend is out of proportion to what it produced, in either direction: a large cost or thinking-token burn for a thin answer, or a near-empty answer at low effort.

For exampleAn agent spends most of its budget reasoning about a simple greeting and returns one sentence.

adaptive_thinking

Split the work up badly

Below release gate

Catches a plan that is broken into subtasks badly: steps that are vague and non-actionable, steps ordered so a prerequisite comes after its dependent, circular or duplicated work, and requirements from the task that no step covers.

For examplePlanned to deploy a service before testing it, and never scheduled the antivirus scanning step at all.

decomposition

Underestimated the work every time

Below release gate

Catches an agent whose pre-execution estimates systematically undershoot what the run actually costs, across time, money, tokens and iterations.

For exampleAn agent schedules a six-step approval process to finish in a few seconds without allowing time for any approvals.

planning_fallacy

Tools, models, and cost1

Picked the wrong worker

Below release gate

Flags LLM calls where the model tier does not fit the work being asked of it: a frontier model spending real money on a trivial call, or a small fast model handling a long, tool-heavy, multi-turn task.

For exampleA workflow assigns a complex legal comparison to a model configured only for simple classification.

model_selection

Truthfulness and disclosure1

Everyone agreed without checking

Below release gate

A multi-agent debate converges because the agents reinforce each other rather than because anyone checked the answer.

For exampleSeveral agents approve a launch plan without addressing a colleague's warning that the payment system is down.

consensus_collapse

n8n3

Built a workflow too tangled to maintain

Below release gate

Flags an n8n workflow whose graph has outgrown what a team can safely change.

For exampleA simple three-node workflow grew to 58 nodes covering five unrelated jobs on one canvas.

n8n_complexity

Left a workflow failure unhandled

Below release gate

Finds places where an n8n workflow hides a failure.

For exampleA failing AI step was set to continue anyway, so a blank summary got written straight to the database.

n8n_errorEvaluation data under review

Let a workflow run without a time limit

Below release gate

Flags an n8n workflow with nothing bounding how long it can run: no credible executionTimeout, HTTP and AI nodes with no timeout, webhook triggers with no credible response timeout, and Merge or Wait nodes in a waiting mode that can block indefinitely.

For exampleWith no timeout set anywhere, a stalled AI call left the workflow hanging for minutes on a support ticket.

n8n_timeout
About these statuses and the evaluation

Production, Beta, Experimental, and Below release gate are Pisama product classifications informed by our latest detector evaluation. Production identifies the strongest-performing group in this benchmark; it is not an independent certification or a guarantee for every workflow.

As of September 21, 2026, this evaluation snapshot covers 105 detectors. We ran the rule-only baseline on 40,108 unique synthetic examples and used separate groups for 1,225 calibration cases and 2,440 evaluation cases. The language-model evaluation starts with Haiku and then uses Sonnet for verification, with response truncation remaining rule-based.

Status bands use conservative results that count unresolved judgments adversely. Evaluation examples were reused across development trials, individual detector samples are small, and known label and input issues remain under review. Delegation has a mismatch between its labels and definition; some workflow-error examples contain malformed inputs. These issues are flagged on their entries.

This directory contains status summaries and examples, not per-example labels, and does not publish the underlying evaluation traces.

Examples on this page illustrate the failure each detector looks for. They are not customer incidents. Performance can vary with the evidence available in your trace, model configuration, and workflow.