Phoenix already has agent evals, what is different?

Name: Pisama
Author: Pisama

Phoenix agent evals are LLM-judge graders that score outputs against a rubric. Pisama detectors run on the trace structure itself: state-hash recurrence catches loops, subsequence matching catches cycles, embedding similarity catches persona drift. These run before the LLM judge ever fires.

Use both. Here's where each one wins.

Observability tools see failures. Pisama acts on them. They are precondition layers; we are the action layer above. The comparison below is honest about where each one is stronger, not a zero-sum claim.

Pisama vs Arize Phoenix

Arize is the strongest pure-play ML observability vendor and Phoenix is its open-source surface. Arize's strength is traditional ML monitoring (drift, performance, embeddings) extended to LLMs with OpenInference traces.

Pisama starts from the agent-failure problem rather than the ML-monitoring problem. The 34-detector taxonomy maps failures to known modes (loops, persona drift, coordination breakdown) rather than statistical anomalies. The two stacks compose well.

Where Arize Phoenix wins

Industrial-strength ML observability lineage and customer base
Embedding drift detection on managed feature stores
OpenInference standard contributor: traces are portable
Phoenix self-host story is mature

Where Pisama wins

57 named detectors covering structural agent failures Phoenix does not target
Heuristic-first pipeline: 90%+ at T1–T3 for free, vs Phoenix's LLM-judge default
Multi-agent coordination as a first-class concept (Phoenix focuses on per-call traces)

At a glance

Dimension	Arize Phoenix	Pisama
Origin	ML observability extended to LLMs	Agent failure taxonomy as ground truth
Trace standard	OpenInference (Phoenix-native)	Ingest OTel `gen_ai.` + OpenInference; emit `pisama.` OTel spans
Detection	Drift + LLM-judge evals	Heuristic + embeddings + LLM-judge tiered pipeline
Multi-agent failures	Trace UI, no named detectors	17 cross-cutting + 6 framework-specific detector packs
License	Elastic 2.0 (Phoenix)	MIT

Recommendation

Phoenix for trace storage, embedding drift, and LLM-judge evals. Pisama for the structural detectors. They emit compatible spans; run both.

FAQ

Phoenix already has agent evals, what is different?: Phoenix agent evals are LLM-judge graders that score outputs against a rubric. Pisama detectors run on the trace structure itself: state-hash recurrence catches loops, subsequence matching catches cycles, embedding similarity catches persona drift. These run before the LLM judge ever fires.