Benchmarks/Detector scoreboard

Detector Calibration Scoreboard

Name: Pisama
Author: Pisama

Every Pisama detector, calibrated on golden datasets and reported per-failure-mode. Hyperscaler agent platforms advertise “anomaly detection” without publishing per-detector F1. We publish the whole table.

Detectors calibrated

53at F1 ≥ 0.80

Mean F1

0.823

range 0.000 – 1.000

Judge cost / full run

$3.66

2,175 LLM escalations

Last calibrated

2026-05-26

14,665 golden samples

	Tier	Mode		95% CI
impersonation_risk	production	heuristic	1.000	1.000–1.000	100.0%	100.0%	85
openclaw_channel_mismatch	production	heuristic	0.993	0.976–1.000	100.0%	98.5%	120
mcp_protocol	production	heuristic	0.990	0.962–1.000	100.0%	98.0%	100
openclaw_session_loop	production	heuristic	0.985	0.959–1.000	100.0%	97.0%	115
task_starvation	production	heuristic	0.980	0.950–1.000	100.0%	96.0%	100
over_refusal	production	heuristic	0.971	0.923–1.000	100.0%	94.4%	85
entity_confusion	production	heuristic	0.968	0.928–1.000	100.0%	93.9%	100
consensus_collapse	production	heuristic	0.967	0.917–1.000	96.7%	96.7%	60
critic_quality	production	heuristic	0.966	0.921–1.000	100.0%	93.5%	95
specification_compliance	production	heuristic	0.966	0.889–1.000	100.0%	93.3%	30
deception	production	heuristic	0.966	0.936–1.000	93.3%	100.0%	85
injection	production	heuristic	0.965	0.935–0.989	94.1%	99.0%	200
under_refusal	production	heuristic	0.964	0.909–1.000	96.4%	96.4%	85
computer_use	production	heuristic	0.960	0.895–1.000	96.0%	96.0%	57
citation	production	heuristic	0.959	0.914–0.992	92.1%	100.0%	115
withholding	production	heuristic	0.957	0.925–0.982	92.5%	99.0%	149
rag_poisoning	production	heuristic	0.940	0.903–0.973	92.2%	95.9%	195
reward_hacking	production	heuristic	0.939	0.889–1.000	95.8%	92.0%	50
openclaw_spawn_chain	production	heuristic	0.933	0.881–0.971	92.7%	94.0%	110
scheduled_task	production	heuristic	0.930	0.868–0.978	100.0%	87.0%	62
dify_variable_leak	production	heuristic	0.923	0.869–0.969	98.2%	87.1%	94
langgraph_tool_failure	production	heuristic	0.920	0.854–0.971	92.0%	92.0%	94
langgraph_parallel_sync	production	heuristic	0.913	0.818–0.982	91.3%	91.3%	97
n8n_resource	production	heuristic	0.909	0.853–0.957	87.0%	95.2%	129
sycophancy	production	heuristic	0.902	0.800–0.979	88.5%	92.0%	50
dify_classifier_drift	production	heuristic	0.900	0.838–0.955	100.0%	81.8%	100
propagation	production	heuristic	0.899	0.833–0.961	100.0%	81.6%	97
jailbreak_compliance	production	heuristic	0.897	0.821–0.984	86.7%	92.9%	85
delegation	production	heuristic	0.895	0.841–0.937	81.0%	100.0%	145
subagent_boundary	production	heuristic	0.894	0.818–0.951	100.0%	80.8%	95
planning_fallacy	production	heuristic	0.892	0.816–0.953	86.1%	92.5%	72
multi_chain	production	heuristic	0.889	0.816–0.944	80.0%	100.0%	63
openclaw_elevated_risk	production	heuristic	0.886	0.830–0.936	79.5%	100.0%	120
scope_escalation	production	heuristic	0.885	0.792–0.957	93.1%	84.4%	85
adaptive_thinking	production	heuristic	0.885	0.830–0.947	79.4%	100.0%	80
openclaw_sandbox_escape	production	heuristic	0.883	0.803–0.920	85.3%	91.4%	120
openclaw_tool_abuse	production	heuristic	0.882	0.807–0.941	97.8%	80.4%	110
orchestration_quality	production	heuristic	0.880	0.766–0.963	84.6%	91.7%	39
n8n_complexity	production	heuristic	0.876	0.821–0.927	81.6%	94.7%	124
role_usurpation_exec	production	heuristic	0.870	0.756–0.960	87.0%	87.0%	83
corruption	production	heuristic	0.869	0.820–0.911	88.9%	85.0%	215
hallucination	production	heuristic	0.866	0.832–0.900	77.0%	99.0%	300
parallel_consistency	production	heuristic	0.860	0.788–0.919	75.4%	100.0%	100
retrieval_quality	production	heuristic	0.858	0.815–0.897	88.4%	83.3%	236
langgraph_state_corruption	production	heuristic	0.846	0.723–0.941	100.0%	73.3%	97
n8n_error	production	heuristic	0.844	0.781–0.901	76.8%	93.8%	127
compaction_quality	production	heuristic	0.844	0.833–0.949	80.7%	88.5%	98
convergence	production	heuristic	0.843	0.761–0.909	86.0%	82.7%	120
context	production	heuristic	0.838	0.809–0.885	87.0%	80.8%	300
dispatch_async	production	heuristic	0.827	0.727–0.911	81.6%	83.8%	86
reasoning_consistency	production	heuristic	0.820	0.727–0.902	83.7%	80.4%	99
memory_staleness	production	heuristic	0.819	0.733–0.889	76.8%	87.8%	99
cowork_safety	production	heuristic	0.800	0.703–0.881	70.2%	93.0%	85
coordination	production-watch	heuristic	0.794	0.746–0.838	84.6%	74.9%	300
langgraph_edge_misroute	production-watch	heuristic	0.793	0.701–0.868	78.6%	80.0%	100
context_precision	production-watch	heuristic	0.789	0.629–0.909	65.2%	100.0%	30
routing	production-watch	heuristic	0.779	0.690–0.851	63.7%	100.0%	100
persona_drift	production-watch	heuristic	0.772	0.711–0.828	76.9%	77.5%	184
multi_agent_contagion	production-watch	heuristic	0.769	0.618–0.881	90.9%	66.7%	60
chunk_relevance	production-watch	heuristic	0.765	0.609–0.919	65.0%	92.9%	28
grounding	production-watch	heuristic	0.758	0.706–0.806	67.6%	86.2%	300
specification	production-watch	heuristic	0.757	0.661–0.835	87.5%	66.7%	300
completion	production-watch	heuristic	0.739	0.690–0.785	67.8%	81.0%	300
loop	production-watch	heuristic	0.732	0.675–0.784	80.9%	66.8%	300
exploration_safety	production-watch	heuristic	0.727	0.593–0.836	80.0%	66.7%	100
authority_gradient	beta	heuristic	0.679	0.513–0.813	81.8%	58.1%	62
model_selection	beta	heuristic	0.675	0.553–0.785	61.9%	74.3%	102
langgraph_checkpoint_corruption	beta	heuristic	0.640	0.527–0.742	50.8%	86.5%	100
derailment	beta	heuristic	0.633	0.557–0.699	52.8%	79.0%	300
communication	beta	heuristic	0.629	0.529–0.716	100.0%	45.9%	149
decomposition	beta	heuristic	0.602	0.537–0.660	45.4%	89.3%	300
overflow	beta	heuristic	0.588	0.487–0.674	81.0%	46.1%	150
approval_bypass	beta	heuristic	0.559	0.457–0.650	88.9%	40.8%	149
workflow	beta	heuristic	0.433	0.345–0.515	74.6%	30.6%	210
role_usurpation_canonical	beta	heuristic	0.425	0.302–0.529	92.3%	27.6%	147
role_usurpation	experimental	heuristic	0.226	0.142–0.307	100.0%	12.7%	263
chunk_attribution	experimental	heuristic	0.000	0.000–0.000	0.0%	0.0%	20

Why publish this

Generic anomaly detection is becoming a feature in every hyperscaler agent platform. That makes “we have anomaly detection” a check-the-box claim, not a differentiator.

Pisama’s position is the opposite: a structured failure taxonomy with calibrated detectors, each tuned and reported on a per-mode basis. Coordination failure does not look like grounding failure does not look like persona drift, and the eval surface should reflect that.

Numbers above are from the Sprint 11 calibration run on golden datasets sized at 14,665 entries. Confidence intervals come from cross-validation. LLM-judge escalation cost is the total spend to recalibrate the entire bench in one pass.

Source code: github.com/Pisama-AI/pisama. Snapshot generated by backend/scripts/generate_scoreboard_snapshot.py.