LLM-as-a-judge best practice is to run many narrow judges, not one big one. Each judge scores one criterion, returns a binary or small-ordinal verdict, and cites the exact verbatim span it judged. Run both answer orders; keep only verdicts that survive the swap. Calibrate each judge against a frozen human-labelled set, re-scored on a schedule, so a changed judge is distinguishable from a worse product.
Ross Jones — Founder, The Hopium Lab. Last modified 22 July 2026. Twenty-four shipped products across fifteen sectors.
Why does one prompt scoring everything fail?
A single prompt weighing eight criteria and emitting one number collapses the trade-off internally, and the collapse leaves nothing to inspect. Ask why an output scored 7 and the only available answer is a paragraph written after the number was chosen.
Decomposition is neither my idea nor new. Anthropic's engineering documentation says to "grade each dimension with an isolated LLM-as-judge rather than using one to grade all dimensions", and Hamel Husain has argued for binary pass/fail plus a written critique over 1–5 scales for a while — rightly, on anecdote across 30-plus companies.
Two mechanisms are missing from almost everything published on judge design. The judge must cite the span. And when a score drops, somebody has to be able to say whether the product got worse or the judge changed.
What is quote-anchoring and what does it buy?
Quote-anchoring is requiring the judge to return the exact verbatim span it is judging, which the harness then verifies is present character-for-character in the source before the verdict is accepted.
Two properties follow, and neither is honesty. A reviewer can confirm or overturn a span-anchored verdict in seconds without re-reading the output. And when a domain expert disagrees, the disagreement lands on a specific string, so the rubric gets fixed instead of argued about. Neither property is free. Spans cost output tokens across every judge, and the harness needs a string-containment check or the judge can invent the quote.
Turpin et al. (arXiv:2305.04388, NeurIPS 2023) showed models systematically fail to mention the features actually driving their outputs, with biasing inputs dropping accuracy by as much as 36% across 13 BIG-Bench Hard tasks while the explanation stayed fluent. A cited span is not proof the span caused the verdict. Quote-anchoring is a workflow property, and workflow properties survive unfaithful reasoning.
Evidence is written before the verdict exists: Multiple Evidence Calibration (Wang et al., ACL 2024) forces evidence ahead of the rating, and two unreviewed 2026 preprints extend the ordering to spans — Rulers (arXiv:2601.08654) and Proof-Before-Preference (arXiv:2605.23970).
A judge that cannot point at the span it is judging has not produced a judgement. It has produced a number you are required to trust.
Which biases are real, and what fixes each one?
Five biases have hard published measurements, and each has a different fix. Rubric wording fixes almost none of them.
| Bias | Sharpest published evidence | What fixes it | What does not |
|---|---|---|---|
| Position | Reordering alone let Vicuna-13B beat ChatGPT on 66 of 80 queries, ChatGPT judging (ACL 2024) | Run both orders, keep verdicts surviving the swap, at 2× cost | Rubric wording. Quote-anchoring. |
| Verbosity | A "repetitive list attack" fooled GPT-3.5 and Claude-v1 91.3% of the time in 2023; robustness 0.884–0.977 on 2024 judges | Statistical length control: length-controlled AlpacaEval moved Spearman against Chatbot Arena 0.94 → 0.98 | Telling the judge to ignore length |
| Self-preference | Reported +10% (GPT-4), +25% (Claude-v1) win rate on own outputs — figures the source paper declines to stand behind | Score against gold judgments (arXiv:2506.02592); treat as a familiarity effect | Swapping judge model. The swap changes whose style wins. |
| Fallacy-oversight | Judges "ignore logical errors in reasoning steps and only focus on the correctness of final results" (CALM, 2024) | A second judge on the reasoning steps, anchored to the faulty step | One judge scoring "correctness" |
| Sentiment | Robustness 0.238–0.307 under fear-laden phrasing — weakest of twelve catalogued biases | Normalise affect before judging; score tone separately | Assuming the judge reads content, not mood |
Position, verbosity and self-preference are the 2023 canon and only a subset. CALM (arXiv:2410.02736) systematises twelve, and for technical content the dangerous ones sit lower down: fallacy-oversight, and authority bias — "the tendency to assign more credibility to statements made by authority figures, regardless of actual evidence". A judge scoring a compliance answer can reward a confident citation of a standard that does not exist.
MT-Bench (arXiv:2306.05685, NeurIPS 2023 Datasets & Benchmarks, v4 December 2023) measured position bias by swapping answers: GPT-4 held its verdict on 65.0% of pairs, GPT-3.5 on 46.2%, Claude-v1 on 23.8%.
The most-quoted numbers in LLM-judge design — GPT-4 favouring its own answers by ten points, Claude-v1 by twenty-five — come from a paper whose own text says it "cannot determine whether the models exhibit a self-enhancement bias".
Wataoka et al. (arXiv:2410.21819) found judges score low-perplexity text higher than humans do "regardless of whether the outputs were self-generated" — familiarity, not vanity, which is why changing judge model does not help.
Should the scale be binary or 1–10?
Binary per criterion. Where degrees are genuinely needed, 0–5. Never a ten-point scale.
The evidence is narrower than the advice, so here is the evidence. The one controlled comparison of grading scales (Li et al., arXiv:2601.03444, 6 January 2026, unreviewed) tested 0–5, 0–10 and 0–100 across six benchmarks and six judges. Pooled human–LLM alignment: 0–5 at ICC 0.853, 0–100 at 0.840, 0–10 at 0.805. Its raters were all graduate students. Binary was not tested, and 0–100 beating 0–10 kills any claim that fewer scale points is monotonically better.
Long ordinal scales compress verdicts into a narrow high band, destroying discrimination between a good answer and an adequate one. Scale reduction is not the only published fix: G-Eval (EMNLP 2023, 2023.emnlp-main.153) takes a probability-weighted sum of scores because raw integer scoring "leads to many ties in the scores", which needs token probabilities most hosted judge APIs no longer expose reliably. The strongest case for binary is not a scale study at all: RLCF (arXiv:2507.18624, NeurIPS 2025) found decomposed checklists beat scalar reward models on every benchmark tested, gaining 4 points of hard satisfaction rate on FollowBench and 6 on InFoBench.
When does decomposition make the judge worse?
When the criterion is a property of the whole output rather than of any span. Zhang (arXiv:2603.28005, 30 March 2026, unreviewed) ran a prompt-controlled comparison and found the holistic judge matched or exceeded the atomic judge on two of three benchmarks, winning on the completeness-heavy sets across all four model families, statistically reliable in three of the four. Proposed mechanism: self-decomposition fragments completeness reasoning across claims, making global omissions harder to see.
So classify the criterion before choosing an architecture. Factual support, citation accuracy, instruction compliance and prohibited content are local, verifiable against a span, and decomposition wins. Completeness, coherence and omission are global — omission has no span by definition.
Who drifted — the product or the judge?
Unanswerable, unless a frozen human-labelled anchor set is re-scored by the current judge on a fixed interleave. The judge sits behind an API, and a silent checkpoint change moves every dashboard without one line of your code changing.
Three drifts get conflated and must not be. Judge-model drift: the model changes under a stable name — Chen, Zaharia and Zou (Harvard Data Science Review, arXiv:2307.09009) recorded GPT-4 moving from 84% to 51% on one task between March and June 2023, and while Narayanan and Kapoor fairly contested the capability reading, behaviour change under an unchanged API name is the problem regardless. Criteria drift: Shankar et al. (UIST 2024, arXiv:2404.12272) watched nine expert users continually refine their standards while grading, revealing "the mutual dependency of criteria on LLM outputs". Input-distribution drift: production traffic walks away from the calibration set.
A single-author preprint (arXiv:2606.15474, 13 June 2026, unreviewed) gives attribution a concrete design — frozen anchor set, a sequential test on the judge-versus-human gap, a verdict in {none, system, judge} — reporting a silent version bump attributed correctly in 60 of 60 runs, against baselines producing 75% false alarms on stable streams. Treat the design as a proposal and the anchor set as the uncontroversial part. Keeping it current is most of the work behind a standing verdict.
How do you know the judge is any good?
Agreement with a human on held-out labelled data, reported as Cohen's kappa plus per-criterion true-positive and true-negative rates. A binary judge is a classifier; reporting Pearson or Spearman for one is a category error.
Report human–human agreement on the same rubric first. A judge at 85% against a rubric two domain experts agree on 70% of the time is not a good judge — it is an unreliable rubric, and prompt engineering will not fix a criterion two people read differently.
One ceiling is proved rather than observed. Dorner, Nastl and Hardt (ICLR 2025, arXiv:2410.13341) show that when the judge is no more accurate than the evaluated model, no debiasing method reduces the required ground-truth labels by more than half. The bound bites at the frontier, not where a frontier judge grades a smaller tuned system. Prediction-powered inference (arXiv:2403.07008) uses a small human-labelled set to estimate and correct the judge's bias, reporting up to 50% gains in effective human-labelled sample size — real, and the same order of magnitude as the Dorner bound.
How do LLM judges fail in production?
In named, recognisable ways. Unnamed failure modes never get budget.
- The mega-prompt. Eight criteria, one number, no way to attribute a drop to any of them.
- The fabricated quote. The judge cites a span absent from the source, and without a string-containment check the fabrication scores.
- No way out. No
INSUFFICIENT_EVIDENCEoption, so a judge lacking information guesses and the guess is recorded as a finding. - The unmeasured rubric. No human–human baseline, so an ambiguous criterion is indistinguishable from a weak judge.
- Silent judge upgrade. A new checkpoint ships under the same model string and a fortnight of eval history quietly stops being comparable.
- Rubric edit, stale labels. A criterion is reworded, the old anchor labels are kept, and the metric silently changes meaning.
- Fallacy-oversight. The final answer is right, the reasoning is wrong, and a correctness judge scores PASS.
Each is cheap to catch before launch and expensive to find after a quarter of eval history has accumulated against a broken judge — which is why judge calibration belongs in the production-readiness categories.
What does the rubric actually look like?
NARROW JUDGE SPEC — one criterion, one judge, one cited span
The Hopium Lab · v1.0 · 22 July 2026 · fork it
# Quote-anchoring is requiring the judge to return the exact verbatim
# span it is judging, which the harness then verifies is present
# character-for-character in the source before the verdict is accepted.
CLASSIFY THE CRITERION FIRST
LOCAL -> verifiable against a span: factual support, citation
accuracy, instruction compliance, prohibited content
-> one narrow judge each
GLOBAL -> property of the whole output: completeness, coherence,
omission -> ONE holistic judge, detailed rubric.
Do not decompose. Omission has no span.
JUDGE OUTPUT SCHEMA (sketch, not literal JSON) — field order is the
contract. Evidence is written before a verdict exists.
{
"criterion_id": "cite_support_v3", # one criterion. No cross-refs.
"evidence": [
{"source_id": "doc_12", "quote": "<verbatim span, <=200 chars>"}
],
"critique": "<=60 words: why the evidence does or does not
satisfy the criterion. Reusable as a few-shot
example.>",
"verdict": "PASS" | "FAIL" | "INSUFFICIENT_EVIDENCE",
"confidence": 0-5 # optional. Never 0-10.
}
HARNESS RULES — enforced in code, never in the prompt
[ ] Reject the run if any quote is not a character-exact substring of
the cited source. A fabricated quote is a failed judge run, not
a FAIL verdict.
[ ] INSUFFICIENT_EVIDENCE always available. A judge with no way out
guesses.
[ ] Pairwise: run both orders, keep only verdicts surviving the swap.
Route every flip to a human.
[ ] Pin the judge to a dated model snapshot id, recorded on every row.
[ ] The judge sees the candidate and the source. Never the author, the
model name, or a previous score.
CALIBRATION HARNESS
[ ] Frozen anchor set, human-labelled, >=100 items per criterion,
stratified over the failure modes that matter.
[ ] Re-score the anchor set with the CURRENT judge on a fixed
interleave -- not only when a dashboard moves.
[ ] Report Cohen's kappa + per-criterion TPR/TNR. Never Pearson,
never Spearman: a binary judge is a classifier.
[ ] Report human-human agreement on the SAME rubric FIRST.
[ ] Version the rubric. A reworded criterion invalidates every anchor
label collected under the previous version.
[ ] Attribution: judge-vs-anchor gap moved -> the judge drifted.
Anchor gap stable, product score moved -> the product regressed.
Diablo, my production-readiness auditor, runs seventeen categories this way — each with its own narrow judge — then a first-party adversarial challenge stage that attacks the findings rather than re-scoring them. Splitting the rubric was the cheap part; keeping the anchor set honest is the standing cost, and it never reaches zero.
Delete the confidence field before deleting the evidence field. A judge that scores without quoting produces opinions at scale, and opinions at scale are what the exercise was supposed to replace.
Ross Jones, Founder, The Hopium Lab. Last modified 22 July 2026. Engineering commentary, not legal advice.