pisamaBack to Pisama

Know what can go wrong.

105 detectors, organized by the failures they catch. Find a category, explore an example, and see the status of every detector.

Detector coverage · Updated September 21, 2026

64Production10Beta11Experimental20Below release gate

Showing 11 of 105 detectors

Coordination and delegation2

Did a teammate's job

Experimental

Catches an agent that crosses into another agent's job: taking over work assigned elsewhere, skipping a required reviewer, or making a binding call that belongs to a different role.

For exampleMid-pipeline, the researcher announced it would 'take over' the implementation, skipping the assigned engineer.

role_usurpation

Team talked past each other

Experimental

Catches teams of agents that fail as a group rather than individually: messages nobody answers, delegation that loops back to where it started, pairs stuck repeating the same request, and conversations that keep going without producing anything.

For exampleTwo agents traded the same 'still waiting' message back and forth for four turns with no progress.

coordination

Grounding and retrieval1

Ignored the sources it found

Experimental

Checks whether the answer actually used the chunks that were retrieved for it, and whether every claim in the answer traces back to at least one of them.

For exampleTold a customer a fix shipped in a version number that appeared in none of the documents it had read.

chunk_attribution

Progress and looping1

Passed a broken result onward

Experimental

Looks across linked workflow runs rather than inside a single one.

For exampleOne branch of a research task stops early, but the final report is assembled as if every branch finished.

multi_chain

Safety and permissions1

Leaked a secret between agents

Experimental

Sensitive content reaches an agent that was never given it.

For exampleA sanitized handoff hid a customer's email, but a downstream agent printed the address anyway.

multi_agent_contagion

Task fidelity2

Answered a different question

Experimental

Catches a data-analysis agent that writes a query which runs cleanly and returns a plausible number, while answering a subtly different question than the one asked.

For exampleAsked for CPU usage during one incident, it averaged the whole month instead and the real spike went unnoticed.

analytical_semantics

Missed a stated requirement

Experimental

Catches the point where a stated intent turns into a working specification and something is lost: a requirement dropped, a constraint negated, a language or count changed, or a spec whose overall meaning has drifted while its vocabulary still matches.

For exampleA rule requiring human review for large refunds got rewritten as automatic, with no review at all.

specification

Tools, models, and cost1

Walked away from its errors

Experimental

Measures whether an agent fixes its own tool errors instead of walking away from them.

For exampleAn agent repeats the same failed tool call without changing its request or trying a recovery step.

tool_error_recovery

Truthfulness and disclosure1

Spoke as someone it wasn't

Experimental

The agent writes as a real person, company, or government agency it has no authority to speak for.

For exampleDrafted outreach signed as a real-sounding employee it was never authorized to speak as.

impersonation_risk

n8n2

Expected information the previous step never sent

Experimental

Catches a node reading fields that the node before it never produced, observed in a recorded execution: undefined field access, property reads on null, and structure that loses fields as the run progresses.

For exampleOne step read a field the previous step never produced, so the run finished 'successfully' with an empty result.

n8n_schema

Put workflow resources at risk

Experimental

Flags n8n workflows that can consume unbounded tokens, items, or time: AI nodes with no credible maxTokens, loop and batch nodes with no credible maxIterations or batchSize, HTTP nodes with no credible timeout, and two or more AI nodes wired directly to one another so each stage feeds the next one the full previous output.

For exampleOne workflow run silently fanned out from one record to 340, costing more tokens than the whole prior month.

n8n_resource
About these statuses and the evaluation

Production, Beta, Experimental, and Below release gate are Pisama product classifications informed by our latest detector evaluation. Production identifies the strongest-performing group in this benchmark; it is not an independent certification or a guarantee for every workflow.

As of September 21, 2026, this evaluation snapshot covers 105 detectors. We ran the rule-only baseline on 40,108 unique synthetic examples and used separate groups for 1,225 calibration cases and 2,440 evaluation cases. The language-model evaluation starts with Haiku and then uses Sonnet for verification, with response truncation remaining rule-based.

Status bands use conservative results that count unresolved judgments adversely. Evaluation examples were reused across development trials, individual detector samples are small, and known label and input issues remain under review. Delegation has a mismatch between its labels and definition; some workflow-error examples contain malformed inputs. These issues are flagged on their entries.

This directory contains status summaries and examples, not per-example labels, and does not publish the underlying evaluation traces.

Examples on this page illustrate the failure each detector looks for. They are not customer incidents. Performance can vary with the evidence available in your trace, model configuration, and workflow.