Every capability in an AI system declares one side before implementation: deterministic or probabilistic. Deterministic if a wrong answer costs money, credentials, an audit finding or a person's trust — there the model may propose but never compute. Probabilistic everywhere else, where a mediocre answer harms nobody. Nothing straddles. A capability that cannot name its side was not designed, only assembled.
Ross Jones — Founder, The Hopium Lab. Last modified 22 July 2026.
The forced choice is not mine. Berkeley's compound AI systems post (Zaharia et al, 18 February 2024) asked whether a system's control logic should be written in ordinary code or driven by a model, then left the question open. Xu et al put a "Symbolic Governor" over a "Probabilistic CPU" in From Craft to Constitution (12 October 2025), then a Probabilistic Processing Unit inside a deterministic kernel in From Craft to Kernel (20 April 2026). Alibaba's Blueprint First, Model Second (August 2025) invokes the model on bounded sub-tasks and never lets it choose the route. Compiled AI (6 April 2026) goes furthest: the model emits executable artefacts once at compile time and never runs again. Elliott Girard, in Towards AI (April 2026), calls the shape deterministic shells and probabilistic cores.
Berkeley asked. Four groups since have answered the same way, reaching for four nouns — CPU, blueprint, compile-time artefact, shell — for one metaphor, largely apart. Convergence is the strongest evidence short of a proof, and the honest response is to credit it rather than restage it as discovery. My addition is smaller and more irritating: the declaration is compulsory, per capability, before implementation, and lives in the repository instead of the prompt.
Which question decides the side?
One question, asked per capability, before anyone opens an editor: does a wrong answer here cost money, credentials, an audit finding, or a person's trust? Yes puts the capability on the deterministic side, where the model may only propose, never compute. No puts it on the probabilistic side, where the model runs free.
Propose has a precise meaning. The model returns a candidate; code validates the candidate against a schema, binds it to a record that already exists, and recomputes every number acted upon. Validation failure queues a human.
The declaration rule: every capability declares one side before implementation, and the declaration is enforced at the boundary rather than requested in a prompt.
Clustering feedback, drafting a first-pass reply, suggesting a tag — a mediocre answer costs nobody anything, so probabilistic is correct and cheap. Computing a refund, granting an entitlement, filing a return, deciding whether a person qualifies — a wrong answer lands on somebody. Deterministic, model demoted to proposing candidates that code checks.
The model may propose. The model may not compute. Anything reaching a ledger, a permission, a filing or a person's inbox is produced by code a human can read.
Why does everyone selling a platform end on a spectrum?
Because a spectrum sells to both buyers and a binary commits to one. deepset published "AI Agents and Deterministic Workflows: A Spectrum, Not a Binary Choice" on 12 May 2025: "neither inherently superior to the other", real implementations occupying positions along a range. Type this page's title into a search box and page one is Diplo, Sombra, Towards Data Science, Latitude, Elementum and a stack of Medium and Substack posts landing in the same place. Zero preprints. The head term is owned by content marketing, and content marketing cannot afford to disqualify a buyer.
An architect holding a capability at ten past nine on a Tuesday needs a side, not a position.
Is the cost test just a risk assessment with better copy?
Partly. Money, credentials, audit findings and trust map onto any competent risk taxonomy, and a credentials failure is usually a money failure wearing a hat. The difference is the output type. A risk assessment emits a score, and a score has a middle. Teams live in the middle, where "mostly deterministic, with an LLM fallback" gets written down and shipped.
A score is where a team goes to avoid a decision. Forcing a binary means somebody has to sign it.
Is anything running on a GPU deterministic anyway?
No, and the explanation most engineers give is wrong. Thinking Machines Lab sent 1,000 identical temperature-zero requests to Qwen3-235B and collected 80 distinct completions, published 10 September 2025. Not concurrent atomic adds and non-deterministic GPU scheduling: the forward pass needs no atomic adds and is run-to-run deterministic inside a fixed batch. Floating-point non-associativity is the enabling condition, never the trigger. The trigger is the batch-size dependence of reduction kernels: dynamic batching changes the reduction tree, so an identical prompt computes differently depending on how many strangers hit the endpoint in the same millisecond. The effect is numerical rather than random, and fixable at a price. Batch-invariant RMSNorm, matmul and attention kernels returned identical output across all 1,000 runs on Qwen3-8B, cutting throughput by roughly 60% in the baseline implementation and by about 34% once SGLang integrated them with CUDA graphs.
So the axis is not determinism in the arithmetic sense. The axis is reconstructability. A capability is deterministic when a human can rebuild, after the fact, exactly what it produced and why, from inputs plus code plus configuration plus the prompt version.
THE REPLAY TEST: Can you rebuild the answer without calling the model? If not, you don't have a system. You have a demo with a try/except around it.
The full test is six boxes and half an hour per capability. Most production systems fail.
What about the capabilities that straddle?
Straddling capabilities do not exist. Every candidate turns out to be two capabilities glued together and never separated: one reads messy input and proposes, one computes and acts. Decomposition is the actual work, and the hedge feels true only because nobody has done it.
| Sold as one capability | Actually two | Probabilistic half — proposes | Deterministic half — computes |
|---|---|---|---|
| Lead scoring | Extraction, then scoring | Reads the transcript, emits typed features | Applies published weights, returns the score |
| Fraud triage | Signal extraction, then rules | Flags anomalous narrative in free text | Thresholds, holds funds, writes the case record |
| Semantic search | Retrieval, then ranking | Produces embedding neighbours as candidates | Filters by entitlement, orders by declared rule, cuts at k |
| Document routing | Classification, then routing | Proposes a label with a confidence | Routes on the label; unroutable goes to a human queue |
| "Agentic" invoice processing | Extraction, then posting | Reads the PDF, emits field candidates | Matches the PO, recomputes the total, posts the entry |
| Agent tool use | Selection, then execution | Chooses the tool and the arguments | Validates arguments, enforces the idempotency key, executes |
Whether the probabilistic half deserves an agent or a plain process step is a separate fork: Agent or Process?
What does a declaration look like in a repository?
One file per capability, checked in beside the code, failing the build when absent. A prompt enforces nothing: a prompt is a request, and a model need not honour a request.
# capability.yaml — one per capability.
# No third value. No "hybrid". No "TBD".
capability: post_invoice_total
side: deterministic # deterministic | probabilistic
wrong_answer_costs: # money | credentials | audit_finding | trust | none
- money
- audit_finding
model_role: propose # none | propose (never "compute")
proposal: # required iff model_role == propose
returns: candidate_fields # typed, schema-validated, never a final value
validated_by: schemas/invoice_v3.json
bound_to: purchase_order_id # must resolve to an existing record
recomputed_by: billing.total() # code recomputes everything acted upon
on_validation_failure: queue_for_human # never: retry until it parses
reproducible_from: [inputs, code_version, config_version, prompt_version]
override: credit_note_with_reason_code # named human override path
stop: halt_before_posting # named safe state
owner: platform-billing
declared: 2026-07-22
# the side check, in full — the rest of the schema is a JSON-Schema job
shopt -s nullglob
files=(capabilities/*.yaml)
(( ${#files[@]} )) || { echo "no capability declarations found"; exit 1; }
for f in "${files[@]}"; do
grep -qE '^side:[[:space:]]+(deterministic|probabilistic)[[:space:]]*(#.*)?$' "$f" \
|| { echo "undeclared: $f"; exit 1; }
grep -qE '^model_role:[[:space:]]+(none|propose)[[:space:]]*(#.*)?$' "$f" \
|| { echo "model_role must be none or propose: $f"; exit 1; }
done
One file and a ten-line loop. I run the identical procedure as a fixed-price go/no-go, and the file above is the whole method — fork it and skip the invoice.
Does the EU AI Act require deterministic components?
No, and a page claiming otherwise has not read the Regulation. Article 15(1) requires an appropriate level of accuracy, robustness and cybersecurity in high-risk systems, and consistent performance across the lifecycle — consistency, not determinism. Article 14(1) requires design permitting effective human oversight; 14(4)(d) the ability to disregard, override or reverse an output; 14(4)(e) interruption into a safe state. Article 15(4) requires resilience to errors, faults and inconsistencies, plus feedback-loop mitigation where a system keeps learning after deployment.
Read the obligations as engineering requirements and the shape appears: overridable, interruptible, consistent, reconstructable. Declaring sides is the cheapest known route to all four — not a mandate, and architecture supports a conformity case rather than conferring one. On timing, the Digital Omnibus deferred stand-alone Annex III high-risk obligations to 2 December 2027 and Annex I embedded systems to 2 August 2028; verify both against the consolidated text. Record-keeping is a separate problem, handled in Article 12 logging.
What does holding the line cost?
Real money and real calendar time. A decision procedure advertised as free reads as marketing. Blueprint First's benchmark gains (arXiv:2508.02721, August 2025) — a 35.56% pass rate against 18.00% for the ATLAS baseline across 1,000 TravelPlanner tasks, and 11 constraint violations against 275 — came from expert-codified blueprints, meaning expensive human design before the first token. Bit-identical inference costs a third to two-thirds of throughput. Decomposing one straddling capability costs a fortnight nobody put in the plan.
The arithmetic on the other side is worse. Carrera and Maldonado-Ruiz measured a ~6.5x latency overhead running simple extraction down the probabilistic path (arXiv:2601.15130, 21 January 2026) and cite regular expressions beating an LLM by up to 28,120x at comparable accuracy — 89.2% against 87.7%. Compiled AI reports a 57x token reduction at 1,000 transactions. Paying a fortnight once to stop paying 6.5x forever is not a close call. Token-bill arithmetic in full: When should you not use an LLM?
How does the rule fail in production?
In seven recognisable ways, all found by reading code rather than architecture diagrams.
- Prompt-as-policy. The rule lives in the system prompt, so the rule is advisory. Models decline advisory rules under load, under adversarial input, and deep into a long context.
- Retry laundering. The wrapper retries until output satisfies the schema, converting a refusal into a plausible fabrication that validates cleanly. Schema conformance is not correctness.
- Validator-as-model. The check on model output is itself a model call, so the deterministic half is probabilistic all the way down. Nobody notices until the check agrees with a hallucination.
- Silent widening. A propose-only capability grows a code path acting on the model's value directly. One new tool, one merged PR, no re-declaration.
- Free-text arithmetic. The model returns a total, a date difference or a pro-rata figure and the code trusts the number. Recompute every number acted upon, including the ones that look right.
- Fallback drift. The deterministic path catches an exception and quietly routes to the model, so the failure mode of the safe side is the unsafe side.
- The undeclared middle. Nobody assigned the capability a side, and each half of the team assumes the other half handled it. Discovered during an incident, always.
Named failure modes get budget; unnamed ones get a retrospective. Diablo, the auditor I run across my own portfolio, checks seventeen production-readiness categories plus a first-party adversarial challenge stage; the undeclared middle recurs most.
Nothing here requires agreement with the declaration rule. The rule requires an answer per capability, written down, before implementation. A team that refuses is not disagreeing. A team that refuses has not decided.
Ross Jones, Founder, The Hopium Lab. Last modified 22 July 2026. Engineering commentary, not legal advice.