Use both. Here's where each one wins.
Observability tools see failures. Pisama acts on them. They are precondition layers; we are the action layer above. The comparison below is honest about where each one is stronger, not a zero-sum claim.
Pisama vs Judgment Labs
Judgment Labs builds judgeval, an open-source agent evaluation SDK, plus Agent Behavior Monitoring for production and the Osiris family of judge models, with Stanford research roots. They raised $32M led by Lightspeed in May 2026. To their credit, they have published a real accuracy number: F1 0.79 for trajectory-hallucination detection against human labels on internal traffic. Most vendors in this category publish nothing.
The comparison with Pisama is breadth and program versus a single number. Pisama ships a full failure taxonomy where every production detector carries a published F1 from a versioned calibration program: open dataset, fingerprints, public changelog. And where Judgment's agent drafts fixes in the prompt editor (their words: nothing persists until accepted), Pisama's healing loop applies the fix server-side and verifies it by re-execution.
- Stanford research pedigree; the Osiris judge models are a real research asset
- Published a genuine accuracy number (trajectory hallucination, F1 0.79 vs human labels), ahead of most of the category on disclosure
- Open-source judgeval SDK with a strong adoption motion
- $32M raised (Lightspeed, May 2026) to push all of it
- 87 named detectors (6 externally validated at production grade) vs a defined-behavior list
- A calibration program vs a single number: per-detector F1 on an open dataset with fingerprints and a changelog; their 0.79 is one metric measured on private internal traffic
- Multi-agent and sub-agent detectors: coordination breakdown, role usurpation, information withholding, silent cascade
- Verified healing: server-side apply, re-execute, re-detect, vs fix drafts in the prompt editor
At a glance
| Dimension | Judgment Labs | Pisama |
|---|---|---|
| Published accuracy | One number: trajectory hallucination F1 0.79, internal data | F1 per detector, open dataset, versioned program |
| Detection breadth | Behavior monitoring on behaviors you define | 87 named failure modes across single, multi, and sub-agent |
| Multi-agent failures | Not targeted as named modes | Coordination, role usurpation, withholding, silent cascade, by name |
| The fix step | Drafts in the prompt editor, accepted by a human | Server-side apply (n8n), re-execute, re-detect |
| Open source | judgeval SDK | MIT detectors and SDK, calibration data published |
Externally validated at production grade: real-trace F1 0.80 or higher, precision 0.70 or higher, 30 or more external traces, external-grounded thresholds, and no per-difficulty blind spot (capability registry, external-only lane, 2026-06-14).
Recommendation
Teams that want to define custom behaviors and own the judging stack will like judgeval, and Judgment's research direction is worth watching. Run Pisama when you want verdicts out of the box: a pre-calibrated taxonomy with per-detector accuracy you can audit before deploying, multi-agent coverage, and a fix loop that verifies by re-execution.
FAQ
- Judgment publishes accuracy too. Does that not neutralize the Pisama claim?
- It narrows it, and we say so. Judgment published one trajectory-hallucination F1 (0.79) measured on internal traffic, which is more than most competitors have done. Pisama publishes F1 for every production detector on an open dataset you can re-run, with dataset fingerprints and a public changelog, including multi-agent failure modes. The question for a buyer is whether the number is auditable and whether it covers the failures you care about.
- Is judgeval a substitute for the Pisama SDK?
- They overlap on single-agent evaluation. judgeval gives you scorers and judge models to build your own evaluation; Pisama ships pre-calibrated detectors with published accuracy plus the healing loop. Build-your-own versus verdicts-out-of-the-box is the real choice, and some teams run both.