Cross-Vendor Judge Agreement: How Pisama Validates Its Labels
Every detector threshold Pisama ships is derived from labeled traces. So the load-bearing question is not "how accurate is the detector" but "where did the labels come from, and would a different judge have produced the same ones?" If ground truth is one model's opinion, every downstream number inherits that model's priors.
Pisama's labeling lane answers this with a panel: external corpora are labeled by judge models from three vendor families (Anthropic, Google, OpenAI), every vendor's verdict is stored per row, and we measure how often the independent judges agree. This note is the public explanation of that methodology.
Very few detectors clear our cross-vendor sample-size publication floor today. On the lane that does, raw agreement between vendor pairs is high while Cohen's kappa reads near zero. Both statements are consistent with each other. The rest of this note is about why a single agreement statistic misleads on a class-imbalanced slice, and what we look at instead.
Method
- External lane only. Synthetic corpora never feed these statistics, the same policy our calibration metrics follow.
- Sample floor. A vendor pair is published only above a minimum usable-row threshold. Below the floor, the pair is listed and its kappa withheld rather than published as noise.
- Abstentions are not agreement. A row counts toward a pair only when both vendors returned a definite verdict. UNCERTAIN abstentions are dropped for that pair and reported separately in the verdict balance.
- Three statistics, read together.Raw agreement (share of identical verdicts), Cohen's kappa (agreement above chance), and positive specific agreement (PSA: of the positive calls either judge makes, the share both make). On an imbalanced slice, raw agreement is carried by the easy negatives, kappa deflates mechanically with prevalence, and PSA tracks the class whose labels actually move thresholds.
- This is judge-vs-judge agreement, not judge-vs-human. Human review enters through a dispute lane: rows where the vendors disagree are extracted with their full traces and adjudicated by hand. It measures whether labels are stable across model families, which is a different (and weaker) claim than agreement with human ground truth.
- Production runs Anthropic models only.The multi-vendor panel exists solely in the labeling and calibration lane, to establish that our labels are not one vendor's priors. We do not run three vendors in production.
Coverage
Coverage is narrow, and we would rather say that than imply otherwise. The hierarchical delegation corpora are labeled by a single judge family, so they have no cross-vendor pair to report; they are listed as excluded rather than silently omitted. Most detector lanes have not yet accumulated enough cross-vendor labeled rows to clear the publication floor at all.
The teaching moment: kappa under low prevalence
The derailment lane went through three labeling generations, and the progression is more informative than any single snapshot.
Three things happened across those generations. First, the OpenAI judge in the early lanes abstained on a meaningful share of rows; a later panel swap nearly eliminated that, restoring three-vendor coverage on rows that had silently degraded to two. Second, the contested rows (any YES vote) were re-adjudicated by hand against the full traces, and most of the disputed positives turned out to be artifacts of feeding the judges truncated text: the large majority of the original consensus positives flipped to negative on full-trace review. That adjudication also produced a doctrine ruling now encoded in the judge prompt: under-delivery is not derailment. Third, with very few genuine positives left in the corpus, kappa and PSA mathematically collapse. Each vendor casts very few YES votes, no two vendors cast them on the same trace, and at that prevalence a single swapped verdict moves kappa far more than a hundred correct negatives would offset.
This is why the underlying artifact reports raw agreement, PSA, and kappa side by side with the verdict balance. At low positive prevalence, raw agreement plus verdict balance carry the interpretation; kappa alone would tell you the judges disagree, which is not what the confusion matrices show. What the slice actually needs is not a better statistic but richer-positive corpora, and that is labeling work in progress, not a number we can publish our way around.
What this supports, and what it does not
- Supported: the labeling rubric for derailment on WildChat produces stable verdicts across three vendor families, with disagreement concentrated in the positive class and resolved by human adjudication on full traces.
- Not supported:"Pisama's detectors are validated across three vendors." One detector, one dataset family, labels only. Coverage is the claim boundary.
- Not supported: any detector accuracy claim. Judge agreement is about label quality, not detector performance, and this note deliberately contains no detector performance numbers.
A full audit of our external label corpora is in progress, applying the adjudication treatment described above to every detector lane. Per-detector performance is intentionally withheld until it completes, because we expect the corrected labels to move it. When the audit lands, what we publish will be the post-audit tier, with the label lineage to back it.
Source artifact: judge_agreement_kappa.json (2026-06-10), regenerable from the per-vendor verdicts stored with every labeled row.