OpenAI vs Anthropic vs Google: same task, same detectors

2-call agentic flow. Phase 1 asks each vendor to produce a 3-step plan. Phase 2 injects that plan back and asks for a structured analysis with consensus, conflicts, gaps, and do_not_relay.

Four source docs. doc_a (foundational facts), doc_b (calibration numbers), doc_c (integrations), and doc_d (trap). doc_d is engineered: it conflicts with doc_b on detector count and F1, and mentions an “unreleased” quantum-resistant detection layer scheduled for 2026 H2. A careful agent should flag the conflict explicitly and list the unreleased claim in do_not_relay rather than repeating it as fact.

13 of 60 detectors. Ten cover the agent’s plan + output (grounding, hallucination, derailment, decomposition, completion, context, injection, workflow, withholding, persona_drift). Three are chunk-level RAG metrics (chunk_relevance, chunk_attribution, context_precision) that exercise the source docs split into ~12 chunks. The remaining 47 detectors in the scoreboard require multi-agent coordination, looping behavior, or framework-specific traces that this fixture does not produce. Honest scope statement: this is not the full Pisama bench, and expanding the bake-off would require a different fixture.

All three vendors produced semantically reasonable outputs. All three caught the numeric conflict between doc_b and doc_d. All three flagged at least one unreleased-feature claim in do_not_relay. To a casual human reviewer the three responses look broadly equivalent.

The detector pipeline does not see them that way. It surfaces three different firing patterns at finer grain than human eyeballing reveals. That is the point worth making to a buyer evaluating cross-vendor agents at scale: identical-looking outputs still produce different detector signatures, and the same independent pipeline catches the differences identically across model providers.

The grounding and hallucination detectors fire across every vendor. That is not a noise problem; both detectors are tuned to flag any wording divergence from canonical sources, including paraphrasing. In production agent systems where citations matter (regulated tenants, audit trails), this is the failure mode that paraphrasing-as-quotation creates. The bundled platform evaluators we are aware of do not surface this distinction.

The detectors that meaningfully differentiated the three vendors on this run varied run-to-run. withholding, decomposition, and context are the ones we have seen flip between vendors most often when re-running this fixture. The asymmetry is real and reproducible. The harness is one command.

withholding: agent leaked information from its internal state that a careful persona should have suppressed. In a regulated context this is the failure mode behind every “agent disclosed non-public information” incident.

persona_drift : agent output is no longer consistent with the assigned role description. Customer-facing agents that start answering off-policy questions trip this.

grounding : agent claims do not align token-for-token with cited sources. Paraphrased citations look right to a human reviewer and wrong to an auditor.

completion: declared subtasks were not all addressed in the output. The classic “model said it would do five things and did three” failure.

workflow : phase-2 execution did not follow the phase-1 plan. The agent told you what it would do, then did something else.

Each of these maps to a real production incident class. The scoreboard publishes the calibrated F1 for each, with confidence intervals. The bundled platform evaluators we are aware of do not.

Proves: the same independent detector pipeline runs end-to-end across OpenAI, Anthropic, and Google on identical input, and produces vendor-distinct firings on a fixture engineered to differentiate behavior. The harness is open source and reruns in seconds for under a cent.

Does not prove: that the bundled platform evaluators (Galileo/Cisco, Gemini Enterprise) miss any of these specific firings on this specific fixture. We did not run their tools side-by-side here; we do not have their licenses. The asymmetry argued in the comparison table above is about what each vendor publishes, which is a different (and weaker) claim than head-to-head benchmark performance.

Why we publish this anyway:the point of an independent evaluator is that it runs everywhere the buyer’s agents run, on the buyer’s definition of failure, with the buyer’s calibration data, on the buyer’s schedule. The bake-off above is a demonstration of the first three properties. The fourth is on the buyer.