pisamaBack to Pisama

Know what can go wrong.

105 detectors, organized by the failures they catch. Find a category, explore an example, and see the status of every detector.

Detector coverage · Updated September 21, 2026

64Production10Beta11Experimental20Below release gate

Showing 10 of 105 detectors

Context, memory, and state1

Scrambled its own records

Beta

Catches agent state that changed in a way the workflow cannot account for: a field that switches type, a status that moves backwards, a value nullified, an identity rewritten, or a reference to an ID that was never seen.

For exampleA tool call silently reset an order's status and erased its tracking ID after it had already shipped.

corruption

Dify1

Exposed private information

Beta

Catches secrets and personal data surfacing in Dify node outputs: API keys, bearer tokens, password and secret assignments, SSN and card number shapes, email addresses, and environment variable references.

For exampleA workflow includes a private API key in its customer-facing response.

dify_variable_leak

Grounding and retrieval2

Looked in the wrong place

Beta

Judges the retrieval step itself rather than the answer: whether the documents fetched were relevant to the query, whether the set covers what the question asked for, and what share of it was usable.

For exampleA search for the current returns policy retrieves outdated pages that do not answer the customer's question.

retrieval_quality

Trusted a tampered document

Beta

Scans the documents a retrieval step returned for injection payloads planted inside them, and checks whether the agent's output went on to reproduce that content.

For exampleA knowledge-base article carried a hidden instruction to leak an admin token, and the agent complied.

rag_poisoning

LangGraph1

Repeated steps without reaching an answer

Beta

The graph is looping without converging.

For exampleA graph keeps alternating between plan and review without ever executing the plan.

langgraph_recursion

Progress and looping1

Passed the buck endlessly

Beta

Reads the handoff spans in a multi-agent run and flags escalation that never resolves.

For exampleBilling sends a refund request to support, and support sends it straight back to billing repeatedly.

escalation_loop

Safety and permissions2

Deleted the only copy

Beta

An autonomous desktop agent takes destructive action against the user's real files.

For exampleA desktop agent deletes the only copy of a shared project folder while trying to tidy up.

cowork_safety

Did something permanent while guessing

Beta

The agent is visibly in trial-and-error mode (saying "let me try", retrying after errors, calling the same tool over and over) and performs an irreversible action anyway.

For exampleWhile guessing how a database tool works, an agent tries a command that permanently deletes records.

exploration_safety

Truthfulness and disclosure2

Didn't correct the boss

Beta

A sub-agent holds evidence that contradicts something the orchestrator already asserted, and softens or drops that contradiction in its reply instead of stating it.

For exampleA researcher finds that the lead agent used the wrong price but agrees with it instead of raising the discrepancy.

authority_gradient

Hid what it knew

Beta

The agent holds relevant information in its internal state and leaves it out of what it passes on.

For exampleTold the release agent 'minor issues' while quietly leaving out three failed health checks and a security warning.

withholding
About these statuses and the evaluation

Production, Beta, Experimental, and Below release gate are Pisama product classifications informed by our latest detector evaluation. Production identifies the strongest-performing group in this benchmark; it is not an independent certification or a guarantee for every workflow.

As of September 21, 2026, this evaluation snapshot covers 105 detectors. We ran the rule-only baseline on 40,108 unique synthetic examples and used separate groups for 1,225 calibration cases and 2,440 evaluation cases. The language-model evaluation starts with Haiku and then uses Sonnet for verification, with response truncation remaining rule-based.

Status bands use conservative results that count unresolved judgments adversely. Evaluation examples were reused across development trials, individual detector samples are small, and known label and input issues remain under review. Delegation has a mismatch between its labels and definition; some workflow-error examples contain malformed inputs. These issues are flagged on their entries.

This directory contains status summaries and examples, not per-example labels, and does not publish the underlying evaluation traces.

Examples on this page illustrate the failure each detector looks for. They are not customer incidents. Performance can vary with the evidence available in your trace, model configuration, and workflow.