You make the record reproducible, not the model. Hash the exact request bytes at the trust boundary before the call, append the provider's verbatim response to an append-only log, and serve every replay from the log instead of the provider. Reproducibility becomes a property of storage rather than of inference — and storage does not get retired, rerouted, or silently reconfigured by somebody else.
Ross Jones — Founder, The Hopium Lab. Last modified 22 July 2026.
Can you still set temperature to zero?
Not on Claude Opus 4.7 and later, or on Sonnet 5. Anthropic's parameter deprecation table marks temperature, top_p and top_k deprecated from Claude Opus 4.7 onward, and a non-default value returns a 400 on Opus 4.7, Opus 4.8 and Sonnet 5. The documented replacement is to omit them and prompt instead. Anthropic's migration guide then closes the argument on the provider's own letterhead: temperature 0 "never guaranteed identical outputs on prior models."
At temperature 0 the sampler is already argmax, so a pinned seed governs nothing — Sara Zan set the interaction out on 24 March 2026. Belt and braces, where the braces do nothing.
Why does the same request return different text?
Because the batch you land in changes between calls. The folk explanation — concurrent GPU kernels racing on floating-point accumulators — names the wrong culprit, and reciting it costs you every reader who has profiled inference. Horace He's Defeating Nondeterminism in LLM Inference (10 September 2025) shows the forward pass requires no atomic adds and is "run-to-run deterministic": identical inputs in an identical batch produce bitwise-identical outputs. Server load moves batch size, batch size moves reduction order inside the kernels, and non-associative floating-point arithmetic makes the reordering visible. Your output depends on how many strangers hit the endpoint in the same millisecond you did.
The measurement, from one prompt and 1,000 completions of Qwen3-235B-A22B-Instruct-2507 at temperature 0: 80 unique completions, the most common occurring 78 times. All 1,000 agreed for 102 tokens — then at token 103, 992 continued "Queens, New York" and 8 continued "New York City".
arXiv:2408.04667 (Atil et al., revised 2 April 2025) puts a number on the damage: accuracy variation up to 15% across runs under nominally deterministic settings, with a best-to-worst gap reaching 70%. arXiv:2506.09501 (Yuan et al., 11 June 2025) measured up to 9% variation and 9,000 tokens of length difference on a distilled reasoning model, from nothing but GPU count, GPU type and batch size.
Hasn't the inference layer already solved non-determinism?
Largely, for people who own the GPUs. vLLM ships batch invariance behind VLLM_BATCH_INVARIANT=1: output independent of batch size and request ordering, NVIDIA compute capability 8.0 or higher, beta, paid for by disabling optimisations like custom all-reduce. Determinism costs latency: Thinking Machines' own workload went from 26 seconds to between 42 and 55 with batch-invariant kernels, and all 1,000 completions came back identical.
Note what the guarantee is pinned to. arXiv:2511.17826 (Zhang et al., November 2025) shows reduction order also moving with the number of tensor-parallel devices, independently of batch size.
Every lever in this section requires owning the servers. Nobody exports an environment variable onto somebody else's fleet. Anthropic exposes no seed parameter at all; OpenAI's seed lives in Chat Completions under a "(mostly) deterministic" hedge and is absent from the Responses API.
Batch invariance is real, shipped, and available only to people who own the hardware. Everybody else is buying inference from a queue whose length they cannot see.
Why can't you just call the model again later?
Because models retire on a schedule and requests past the retirement date fail. Anthropic commits to at least 60 days' notice and publishes the dates: claude-3-7-sonnet-20250219 and claude-3-5-haiku-20241022 retired 19 February 2026, claude-3-haiku-20240307 on 20 April 2026, claude-sonnet-4-20250514 and claude-opus-4-20250514 on 15 June 2026.
Set the retirement schedule against the retention obligation. EU AI Act Article 19(1) will require providers of high-risk systems to keep the Article 12(1) logs for a period appropriate to the intended purpose and at least six months, to the extent the logs are under their control and unless Union or national law — data protection law in particular — says otherwise; Article 26(6) puts the same six-month floor on deployers for the logs they hold. Stand-alone Annex III obligations apply from 2 December 2027, Annex I embedded systems from 2 August 2028. Sixty days of notice against a six-month floor is a structural mismatch, not a rounding error: the obligation outlives the thing that produced the record. Article 12 asks for logging capability, not reproducibility; the mechanics are in Article 12 logging.
A reproducibility strategy that ends in a call to the provider has an expiry date, and the provider sets it.
What does each layer actually buy you?
Four of the five layers buy you nothing you can hold onto, because four of them belong to whoever owns the GPUs.
| Layer | Controls | Available to | Survives retirement | Guarantee |
|---|---|---|---|---|
temperature / top_p / top_k | Sampler shape | Nobody on Opus 4.7+ or Sonnet 5 | n/a | None, and never did |
seed | Sampler PRNG above temperature 0 | OpenAI Chat Completions; no Claude equivalent | No | "(mostly) deterministic" |
VLLM_BATCH_INVARIANT=1 | Reduction order across batch sizes | Self-hosters, NVIDIA CC 8.0+, materially slower | While you hold the weights | Bitwise, at fixed GPU type, TP size and version |
| Pinned weights, frozen serving stack | Everything above | Self-hosters paying to freeze for years | Yes | Strong, and expensive |
| Recorded response in an append-only log | Nothing about inference | Anyone, API consumers included | Yes | Byte-for-byte, for the life of the storage |
Only the bottom row is available to an API consumer, and only the bottom row holds when somebody else changes infrastructure without telling you.
Where does a record's identity come from?
From bytes hashed before the provider ever saw them. Pre-call identity is the content hash of the exact bytes that will be sent to the provider, computed at the trust boundary before the call is made, and stored on the record as an immutable content key.
Compute identity after the call and the record is keyed on a response nobody can reproduce. Specify SHA-256 explicitly: Git shipped an experimental SHA-256 object format in 2.29 in October 2020 and is still mid-transition, which tells you how long a hash choice lives. Domain-separate the digests so a request hash can never be read as a response hash — a two-line fix with its own post. The minimum a record must contain is set out in the Replay Test; schema, ordering and the replay path start where that list ends.
What did Fowler write about replaying against external systems in 2005?
That replaying a log against an external system corrupts it, because the external system cannot tell a replay from real processing — and that the fix is a Gateway. Event Sourcing (12 December 2005) captures every change as a sequence of events and rebuilds state by re-running the log.
The LLM provider is Fowler's external system, described two decades before it existed. Replay serves the recorded response and never reaches the network: the gateway in replay mode returns stored bytes or raises. A replay path that falls back to the provider is a cache with good manners.
Yohei Nakajima's The Log is the Agent (arXiv:2605.21997, 21 May 2026) makes the log the source of truth and the working graph a deterministic projection, with replay that reconstructs a run without re-execution. Nakajima is building an agent runtime; a record built as an evidence layer answers to a hostile reader months later, without your codebase in the room.
Does a reproducible record need CQRS?
No, and reaching for CQRS by default is a tell. Fowler's note on the pattern (14 July 2011, crediting Greg Young) opens by telling readers to be very cautious: command-query separation bolted onto a system that reads the way it writes adds significant complexity.
One case earns it: the write side is the immutable evidence log, and every queryable surface is a projection you can drop and rebuild without touching the record. Everywhere else an append-only table plus a refreshed read model is enough. Command-query separation and event sourcing are separate patterns that compose; conflating them is the other tell.
Do idempotency keys give you reproducibility?
No. Different guarantee, different lifetime, different storage. An idempotency key buys exactly-once effect inside a retry window — Stripe prunes keys once they are at least 24 hours old, after which a reused key generates a fresh request. Key derivation is covered separately; note only that draft-ietf-httpapi-idempotency-key-header expired at revision -07 on 18 April 2026 and never became an RFC. Cite it as expired or not at all.
Idempotency buys exactly-once effect for a window measured in hours. Reproducibility buys a reconstructable record for years. Filing the two as the same class of control is how a system ends up with neither.
How do you know the replay path still works?
You run a divergence harness on a schedule. A replay path nobody exercises is a replay path that broke in April and told no one. A divergence harness is the scheduled job that rebuilds stored records from the log, asserts the reconstruction is byte-identical, and separately re-runs the same request live to measure how far today's provider output sits from the recorded one. Not calibration in the ML sense — no expected calibration error, no Brier score.
Two assertions, one of which is allowed to fail. Reconstruction must be exact. Live divergence is information, not a defect: when the gap jumps, somebody changed something they never announced — which is why OpenAI ships system_fingerprint at all.
Six named failure modes:
- Replay-by-re-call. The replay path reaches the provider on a cache miss. Green tests, no evidential value, dead on the retirement date.
- Hash-after-normalisation. Identity computed on cleaned input; the record commits to bytes that never arrived.
- The retirement wall. Every stored record names a model that no longer answers, discovered the day somebody asks.
- Silent provider drift. Backend configuration moved, no version string moved with it, and the divergence is unattributable.
- The unrun harness. Replay works the day it ships and is never run again before an incident review.
- Idempotency mistaken for reproducibility. Keys pruned at 24 hours, obligations measured in months.
What do you actually build?
An append-only record with ordering as a field, a gateway that cannot call out in replay mode, and a harness on a cron.
THE REPRODUCIBLE RECORD — schema and replay contract
The Hopium Lab · v1.0 · 22 July 2026
Supports a conformity case. Does not confer one.
# Pre-call identity is the content hash of the exact bytes that will be
# sent to the provider, computed at the trust boundary before the call is
# made, and stored on the record as an immutable content key.
record:
schema_version: "1.0" # on every record, always
event_id: uuidv7 (RFC 9562) # one per call, always
content_key: sha256(0x00 || request_bytes) # the serialised HTTP body
seq: monotonic, gapless, per stream # ordering is a field
occurred_at: RFC 3339, UTC, written by the writer
input_digest: sha256(0x02 || input_as_received) # pre-normalisation
prompt_digest: sha256(0x03 || prompt_bytes) # never a template name
retrieval: [{doc_id, doc_version, chunk_digest}] # as retrieved
provider: name, endpoint, api_version
model_id: exact string, e.g. claude-opus-4-1-20250805
(retiring 5 Aug 2026 — that is the whole point)
params: object as sent (empty on Opus 4.7+ — 400 if you set one)
fingerprint: provider backend id where exposed, else null
response_bytes: verbatim, committed BEFORE redaction or formatting
# redaction is a projection, never a mutation of the record
# personal data in response_bytes is a retention question:
# Art 19(1)'s six-month floor yields to data protection law
# streamed: concatenated event payloads in arrival order
response_digest: sha256(0x01 || response_bytes) # distinct domain prefix
error_bytes: verbatim provider error body when outcome != COMMITTED
outcome: COMMITTED | FAILED | UNKNOWN
REPLAY CONTRACT
[ ] Replay mode has no code path reaching the provider. Assert on the socket.
[ ] Reconstruction is byte-identical or the replay fails. No fuzzy matching.
[ ] content_key computed before the call, never recomputed afterwards.
[ ] Digest domains stay distinct: 0x00 request, 0x01 response, 0x02 input,
0x03 prompt.
[ ] seq is assigned by a single writer, or by a counter incremented in the
same transaction as the insert. Database sequences are NOT gapless —
rollbacks and caching burn values. A gap fails the audit: a gap is a
lost record or a deleted one.
[ ] Retention outlives the model. Nothing re-calls a provider for evidence.
[ ] The divergence harness runs weekly and its failures page a human.
[ ] A reader outside the team can rebuild a record from the log and this
schema alone, with no access to your codebase.
Delete the "no code path reaching the provider" line and the record quietly reverts to a cache. Enforce it in the transport layer, not in a comment: comments do not survive a refactor eighteen months out; a blocked socket does.
One question settles it: can you rebuild the answer without calling the model? A record you cannot rebuild without the provider is not a record. It is a bet that the provider outlives your obligation. Storage is the one layer of this stack you will still own in three years. Put the evidence there.
Ross Jones, Founder, The Hopium Lab. Last modified 22 July 2026. Engineering commentary, not legal advice.