Replay a frozen corpus of hand-reviewed cases through both prompt versions, score every case per rubric, and diff the rubrics separately. Block the ship on named critical categories only. One rubric can invert while the total holds flat. A dashboard reports that flat total as no change. A flat total is not evidence that nothing moved.
Ross Jones — Founder, The Hopium Lab. Last modified 22 July 2026. Twenty-four shipped products across fifteen sectors.
A prompt-regression harness is a frozen corpus of human-reviewed cases, replayed under both prompt versions, scored per rubric rather than in aggregate, with a hard ship-block on named critical categories. Distinct from the calibration harness, which measures whether a judge still agrees with a human. This one measures whether a prompt still passes cases it used to pass.
Prior art first. Emmimal P Alexander published the same shape on Towards Data Science on 29 June 2026: frozen corpus, forty golden queries, a ship-block on critical categories, and a name for the pattern — False Improvement, the aggregate rising while a category collapses underneath. I am not claiming the shape. What it leaves open: corpus size, which test that size earns, what stops the corpus rotting, whether to buy any of it.
Why does an aggregate score hide a broken prompt?
Because an aggregate is a net figure, and net figures cancel.
The failure has a name already. Yan et al. at Amazon AI (arXiv:2011.09161, CVPR 2021) named the negative flip: the new model gets a case wrong that the old one got right. Image classification, not prompting — prior art the language-model field is busy re-deriving under worse names. Cutting overall error and cutting negative flips are separate objectives.
A 2026 preprint puts language-model numbers on the effect. arXiv:2604.27405 (30 April 2026, single author, not peer reviewed) replayed 2,000 MMLU-Pro items across model upgrades. Llama 3 to 3.1 gained 1.6 points overall: 34% of items improved, 28% deteriorated. Qwen 2.5 to 3 gained 2.8 points, 47% improved, 39% deteriorated. Llama lost physics; Qwen lost law. A release note reading "+1.6" describes a corpus where a quarter of the cases went backwards. Prompt edits invert categories the same way.
An aggregate score is what remains after improvements and regressions cancel. A flat total can mean nothing moved, or that everything moved in opposite directions.
How many cases do you actually need?
Forty to sixty for a ship-gate, with no critical bucket under eight. That detects a collapse in a named category; it cannot detect a drift. It also sits below the tier most guidance labels minimum viable for production decisions — a property of the instrument, not a defect in it. Guidance spans an order of magnitude because almost none of it says which of the two it aims at.
| Corpus size | Source of the number | What it resolves | What it cannot do |
|---|---|---|---|
| 40–50 | Eval-docs starter guidance; the TDS suite uses 40 | One named category collapsing | Any drift claim |
| 50–100 | Same tiering, "minimum viable for production decisions" | Effects of 5–10 points | Per-rubric claims |
| 300–500+ | FutureAGI's stated per-route floor | Effects of 1–3 points | Getting written by a team of three |
Pairing is what makes a small corpus work. Every case runs through both versions, so the comparison is within-case and the variance from harder cases drops out. Anthropic's "Adding Error Bars to Evals" (arXiv:2411.00640, 19 November 2024) recommends paired differences for exactly that reason.
Paired binary outcomes have a correct test, and a percentage-point threshold is not it. Run McNemar's, exact binomial form, on the discordant pairs — cases that passed under one version and failed under the other. Power depends on the count of discordant pairs, not on corpus size. Below roughly 25 the chi-squared approximation stops being trustworthy; a 40-case corpus sits below 25 for anything short of a rewrite.
Now the concession, before a statistician makes it for you. A six-case bucket cannot gate anything: one flip moves it 16.7 points, and one-sided exact McNemar cannot reach p ≤ 0.05 until five of the six have flipped. Critical tags get eight cases minimum, or they report rather than block. Buckets are clusters, too: Anthropic's eval statistics paper warns clustered standard errors on popular evals can run over three times the naive figure.
A ship-block should fire on collapse, not on drift. Forty cases can evidence a collapse. A gate that pretends they evidence a drift teaches everybody to override it.
What goes into the frozen corpus?
Cases written by hand because something went wrong, not cases sampled to represent your traffic.
Harvest from incidents: complaints, human overrides, refusal boundaries, negation, injection attempts, the input three orders of magnitude longer than the median, the empty field. Every case stores expected properties, not an expected string — schema validity, a required extraction, a forbidden phrase, a mandatory refusal. Human-reviewed means a named person signed those properties, dated, in the file.
The certification-retrieval line I run — FAR Part 25 and CS-25 lookups, where a confidently wrong clause reference is the risk — started as hand-written refusal cases and grew every time a reviewer overruled the system. Corpora grown from planning meetings test only the failures you already imagined.
How do you diff per rubric without measuring judge noise?
Replay every case at least five times per version, collapse each case to one verdict by majority, and split the rubrics into deterministic checks and model-graded ones. Only the deterministic checks may block a ship.
One run per case is an anecdote. The 2026 MMLU-Pro preprint reports single-shot greedy evaluation missing 42% of items that had reliably changed and falsely flagging 25% that had not. Five is the smallest odd n where a majority survives one anomalous run: a floor chosen for cost, not a power calculation, and stated as such. The case, never the replay, is the unit of analysis — replays inside a case are not independent.
Replay under production sampling settings, not an artificially pinned decode. On current frontier models temperature is often not available to pin anyway — the reproducibility problem, not a defect in the harness. Record model id and snapshot on every run row, so a provider-side change stays distinguishable from a prompt edit.
Model-graded rubrics have a hard ceiling. Zheng et al. (arXiv:2306.05685, NeurIPS 2023 Datasets and Benchmarks) found strong judges agreeing with human preference at over 80% — roughly the rate humans agree with each other — while naming position, verbosity and self-enhancement biases. A judge at that ceiling cannot resolve a three-point per-rubric delta. Model-graded rubrics report; they gate only where that judge's agreement rate has been measured and written down — its own discipline, not a line item in a ship-gate.
The Towards Data Science suite sidesteps judge noise with a deterministic simulator rather than live model calls — defensible, but its published figures are simulated, not measured. Surviving real provider non-determinism is the harder artefact, and the one a deploy should rest on. Replay also assumes the record was stored properly, which is the Replay Test one layer down.
What blocks a ship, and what only gets reported?
A named critical category losing more cases than it gains, on deterministic checks, at one-sided exact p ≤ 0.05, blocks the deploy. Everything else reports.
Say the arithmetic out loud, because a gate is only as honest as its own power. One-sided exact McNemar needs five discordant pairs pointing the same way to clear 0.05. The False Improvement case in the Towards Data Science write-up — a negation category falling from 100% to 33.3% on six cases, four flips — comes out at one-sided p = 0.0625 and would not fire. That argues for eight-case minimum buckets, not a softer threshold.
Keep the critical list to four rubric tags at most, named in the repository and changed only by pull request, so widening the gate leaves a trace with a name on it. Four tags each tested at 0.05 is a family-wise error rate near 0.19 — the honest reason for the cap. The gate is built to over-block; four is where over-blocking stops being tolerable. An improved aggregate is not a defence: the premise of the gate is that the aggregate improved.
Allow an override, and require a written reason stored beside the run. A gate with no override gets deleted within a quarter. A gate whose overrides are invisible gets used every Friday afternoon. Same reasoning as Article 12 logging: the decision matters less than being able to see who made it. I apply the identical rule reviewing other people's systems in a go/no-go.
Doesn't a frozen corpus rot?
Yes — every pull request run against the corpus is another query to a holdout, and adaptive reuse overfits a holdout.
Dwork et al. formalised the effect (arXiv:1506.02629, NIPS 2015): reuse a holdout adaptively and it stops generalising. For balance, Recht, Roelofs, Schmidt and Shankar's replications (arXiv:1806.00451 CIFAR-10, 2018; arXiv:1902.10811 ImageNet, ICML 2019) found observed adaptive overfitting limited to non-existent. Formally real, empirically milder than feared — which argues for a cheap policy, not alarm.
Three rules cover corpus rot. Seal a slice of cases that iteration never touches, opened on a schedule. Re-sample from production traces quarterly. Never edit a case because it failed — the named failure mode is fixing the test, and every team does it once.
Should you buy an eval platform?
Not until you have written the cases by hand, because no platform can tell you what your own failure modes are, and the corpus outlives every tool you put it in. Hamel Husain put the general form plainly on 29 March 2024: "Keep it simple. Don't buy fancy LLM tools." He means build before buy; the same post insists on looking at enormous quantities of data.
Vendor mortality is the second half. Humanloop shut down on 8 September 2025 after Anthropic hired its founders and most of its engineers. Promptfoo agreed to be acquired by OpenAI on 9 March 2026 — Promptfoo commits to maintaining the open-source suite, while saying nothing about licence terms, self-hosting, or whether the CLI stays free. Two of the most-recommended eval platforms changed hands or died inside six months of each other.
This is not an anti-tooling argument. Promptfoo, DeepEval, Langfuse and Inspect AI from the UK AI Security Institute are free and self-hostable, and I run several. The objection is narrower: do not outsource the corpus.
The corpus is the asset. An eval platform is a rental agreement on your own test data.
What does the prompt-regression harness actually look like?
A directory, a runner, and seven rules.
PROMPT-REGRESSION HARNESS — structure and gate rules
The Hopium Lab · v1.0 · 22 July 2026
corpus/
cases/<id>.yaml input, rubric_tags[], criticality, provenance_date,
reviewer, expected_properties[] (properties, not strings)
sealed/ never used for iteration; opened on a schedule only
runs/
<prompt_sha>/<case_id>/<replay_n>.json
raw output, model id, model snapshot, timestamp, cost
report/
per_rubric.md per tag: pass counts, discordant pairs (b, c),
one-sided exact p, verdict, aggregate reported LAST
RULES
1. n >= 5 replays per case per prompt version. Collapse to one binary per
case per version by majority (>= 3 of 5 passing = PASS). The case, not
the replay, is the unit of analysis; replays are not independent.
2. Deterministic checks (schema, forbidden phrase, extraction match, tool
choice) may block a ship. Model-graded rubrics report only, unless that
judge's agreement with a human has been measured and written down.
3. Compare paired: same case, both versions, McNemar exact binomial on the
discordant pairs. Never a percentage-point threshold on the totals.
4. CRITICAL = at most four rubric tags, named in the repo, changed by PR.
Four tags at 0.05 each is a family-wise error rate near 0.19.
5. SHIP-BLOCK: any CRITICAL tag whose new-fails exceed new-passes at
one-sided exact p <= 0.05 blocks the deploy, whatever the aggregate did.
One-sided exact needs >= 5 discordant pairs in one direction, so no
CRITICAL tag may hold fewer than 8 cases. A 6-case bucket cannot gate
anything and must not be marked CRITICAL.
6. Override is allowed. Override without a written reason stored next to
the run is not.
7. Never edit a case because it failed. Add a case. Editing the test is
the failure mode, not the fix.
QUARTERLY
Re-sample cases from production traces. Retire cases now covered
elsewhere. Open one sealed slice and compare against the live corpus.
Rule 1 sets the bill: five replays across sixty cases and two versions is 600 calls per pull request, which is why it runs per pull request, not per commit.
Write the corpus before you need it. The morning you need it is the morning after a tweak that lifted the score and quietly broke the one category nobody had a rubric for. Forty hand-written cases you can replay beat every dashboard you were paying for.
Engineering commentary, not legal advice. The harness above is deliberately boring; the interesting part is the cases only you can write.