The Replay Test
Fifteen days to find out, capability by capability, whether you have an AI system or a demo with a try/except around it.
The problem it solves
The pilot demoed beautifully and then met real traffic. Nobody can tell you which parts are supposed to be probabilistic and which are quietly guessing at things that must never be guessed.
Who buys it: Head of Data, Head of Digital, CTO or Head of Quality and Compliance at a 200–10,000 person company in a regulated or asset-heavy sector.
What lands
- The capability register: every AI-touching capability declared deterministic, probabilistic, or must-be-rebuilt — with the reason written out. Nothing comes back as "it depends".
- The replay verdict: for each capability, can the answer be rebuilt without calling the model? Pass, fail, or not yet.
- The line, drawn: what the model may propose, and what a computer must compute. Nothing straddles.
- Named failure modes with the specific inputs that trigger them.
- A sequenced remediation plan, with the capabilities that must be rebuilt separated from the ones that need a boundary.
- Merged code where the boundary is cheap to enforce.
What this is not
- No model training, fine-tuning or evaluation-harness build.
- No full rebuild of the capabilities marked must-be-rebuilt — that is quoted separately once the register exists.
- No legal opinion on regulatory status.
Who shouldn't buy it: Teams still deciding whether to use AI at all. This tests a system that exists, against traffic it has already seen.
The guarantee
Every capability in the register gets a declaration and a reason. If any capability comes back undeclared, that section is refunded.
Why me
The Replay Test is the question I apply to my own systems before they carry anything that matters, across twenty-four shipped products in regulated and consumer contexts.
Questions people actually ask
What is the Replay Test?
Can you rebuild the answer without calling the model? If not, you don't have a system. You have a demo with a try/except around it.
What if everything passes?
Then you get a short report saying so and you have bought certainty, which is what you were actually buying. It has not happened yet.