Detectors

Failure detectors across single-agent, multi-agent, and sub-agent systems

Detectors in the capability registry are measured on the external-only lane: real traces from public benchmarks and production integrations. 20 are externally validated at production grade, with per-detector readiness published below.

Production grade
20

Externally validated at production grade against real external traces, not synthetic-only calibration data.

Agent core

  • communication
    Experimental
    Agent A sends information; agent B never acknowledges or references it.
  • completion
    Failing gate
    Subtask coverage falls short of declared success criteria.
  • context
    Beta
    Upstream context entities never referenced downstream.
  • coordination
    Failing gate
    Cross-agent entity reference rate below threshold.
  • corruption
    Experimental
    Shared state schema or types mutate mid-run.
  • derailment
    Failing gate
    Agent output drifts from task definition mid-run.
  • hallucination
    Beta
    Output contains content unsupported by sources.
  • injection
    Production
    Prompt injection or instruction override patterns in input.
  • loop
    Experimental
    State recurrence within a turn window.
  • persona_drift
    Experimental
    Agent output diverges from declared role, tone, or scope.
  • withholding
    Failing gate
    Agent has answer in working state but does not surface it.

Retrieval and grounding

  • chunk_attribution
    Beta
    Source chunks for cited claims mis-attributed or missing.
  • chunk_relevance
    Beta
    Retrieved chunks irrelevant to the query.
  • context_precision
    Beta
    Precision of retrieved context against query intent.
  • grounding
    Beta
    Output entities lack source attribution.
  • rag_poisoning
    Production
    Adversarial content in retrieval corpus affecting output.

Reasoning and planning

  • convergence
    Production
    Metric-aware: trajectory slope, regression, divergence on cost or accuracy.
  • decomposition
    Failing gate
    Planner breaks task into subtasks that miss requirements.
  • specification
    Beta
    Agent interprets task differently from user intent.
  • specification_compliance
    Production
    AgentPex pattern (Sharma et al., 2026). Extracts behavioral rules from system prompts and checks trace for compliance.

Orchestration

  • delegation
    Beta
    Delegation chain loses context or authority.
  • redundant_delegation_conflict
    Experimental
    Duplicate delegations produce conflicting results that go unreconciled.
  • routing
    Production
    Request routed to wrong specialist or wrong model.
  • silent_cascade
    Production
    Sub-agent failure propagates upward without surfacing to the parent.
  • synthesis_failure
    Production
    Parent agent mis-synthesizes or drops sub-agent results.

n8n

  • n8n_complexity
    Failing gate
    n8n workflow complexity exceeds maintainable threshold.
  • n8n_error
    Experimental
    n8n execution error pattern matched to known failure mode.
  • n8n_resource
    Failing gate
    n8n node accesses or mutates an unauthorized resource.

OpenClaw

  • openclaw_elevated_risk
    Experimental
    OpenClaw session enters elevated-risk control state.
  • openclaw_sandbox_escape
    Beta
    Sandbox isolation boundary violation in OpenClaw runtime.
  • openclaw_spawn_chain
    Production
    Agent spawn chain exceeds depth or fan-out limit.
  • openclaw_tool_abuse
    Production
    OpenClaw tool invocation pattern matches abuse signature.

Other

  • analytical_semantics
    Production
    Categorization in next refresh.
  • impersonation_risk
    Production
    Categorization in next refresh.
  • jailbreak_compliance
    Experimental
    Categorization in next refresh.
  • multi_agent_contagion
    Production
    Categorization in next refresh.
  • n8n_cycle
    Failing gate
    Categorization in next refresh.
  • n8n_schema
    Failing gate
    Categorization in next refresh.
  • n8n_timeout
    Failing gate
    Categorization in next refresh.
  • openclaw_session_loop
    Production
    Categorization in next refresh.
  • output_validation
    Production
    Categorization in next refresh.
  • over_refusal
    Production
    Categorization in next refresh.
  • reward_hacking
    Production
    Categorization in next refresh.
  • role_usurpation
    Production
    Categorization in next refresh.
  • role_usurpation_canonical
    Production
    Categorization in next refresh.
  • role_usurpation_exec
    Production
    Categorization in next refresh.
  • scope_escalation
    Production
    Categorization in next refresh.
  • under_refusal
    Beta
    Categorization in next refresh.

Methodology

Each detector is calibrated on the external-only lane: real agent traces from public benchmarks (MAST, GAIA, and others) and production integrations, with detector-level ground-truth labels. Calibration is cross-validated with per-difficulty stratification (easy / medium / hard) per the Anthropic Demystifying Evals methodology.

The scoreboard above is dated 2026-08-21, from the capability registry. We recalibrate per sprint and version the report. Sycophancy, consensus_collapse, and specification_compliance are measured rows in that scoreboard.

Detection is tiered. Tier 1 hash and delta detectors run at zero cost in under ten milliseconds. Tier 2 embedding detectors run at near-zero cost. Tier 3 LLM judges handle genuinely ambiguous cases. Tier 4 human review handles the residue.

MIT core, source-available detectors. The SDK is MIT licensed; the specialised detector library ships under the Business Source License 1.1, which is source-available rather than open source. Both live at github.com/Pisama-AI, and the calibration dataset is published with the detector code.

What this enables

  • Pick detectors by readiness tier (production, beta, experimental) rather than running everything blindly.
  • Audit calibration: every readiness tier is reproducible against the published dataset.
  • Route at the right cost tier: a hash-recurrence detector does not need an LLM judge fallback.
  • Compare detectors across releases: per-detector readiness lets you spot regressions before they ship.
© 2026 Pisama · San Francisco