You validate an AI model under GxP by putting it behind a deterministic decision boundary: qualify it as a computerised system — Annex 11 and draft Annex 22 in EU GMP manufacture, 21 CFR 210/211 in the US — treat its output as evidence, not the decision of record, hold model, system and process under change control, and leave the disposition to a named human.
Ross Jones — Founder, The Hopium Lab. Last modified 22 July 2026.
Engineering guidance, not regulatory advice. Every clause number below comes from a draft and must be checked against the adopted text. In the EU the Qualified Person certifies the batch under Annex 16; in the US the quality control unit approves or rejects under 21 CFR 211.22(a). Neither role can hand certification to a model.
What is the current status of Annex 11 and Annex 22?
Annex 11 has been in force since 2011. The revision is a draft, nothing new has been adopted, and anyone writing that Annex 11 "has been revised" is describing a consultation document.
On 7 July 2025 the European Commission and PIC/S jointly published a revised Chapter 4, a revised Annex 11, and an entirely new Annex 22 on artificial intelligence; consultation closed 7 October 2025. The draft grows Annex 11 from five pages to nineteen and contains no AI-specific requirement whatsoever. Annex 22 is six pages and carries the lot.
Consultation comments are still being worked through, with final text widely expected in 2026 and an operative date after that. EMA ran a workshop on 30 June and 1 July 2026 to test whether generative AI can be enabled at all. Drafting was joint with PIC/S, so the same requirements reach fifty-plus authorities through PE 009. Build to the draft anyway: architecture takes longer to change than guidance does.
Which instrument actually governs your model?
In the EU, Annex 11 today and Annex 22 once adopted; in the US, 21 CFR 210/211 with Part 11 over the records. Everything else in the table is either out of scope or not law, and citing the wrong one is the fastest way to lose an argument with an inspector.
| Instrument | Status, July 2026 | Reaches | Does not reach |
|---|---|---|---|
| EU GMP Annex 11 | 2011 text in force; 2025 revision not adopted | Computerised systems in GMP manufacture; EU and PIC/S | Zero AI-specific clauses |
| EU GMP Annex 22 (new, draft) | Not adopted | AI in critical GMP applications — medicines, active substances | GCP, GLP, GVP, GDP. Not "GxP" |
| 21 CFR 210/211 | In force | US drug cGMP; quality unit approves or rejects | No AI text; predicate rules only |
| 21 CFR Part 11 | In force; unchanged in substance since 1997 (last amended 88 FR 13018, 2 March 2023) | Electronic records and signatures | Says nothing about models |
| FDA Computer Software Assurance | Final, 24 September 2025 | Device production and quality system software, 21 CFR Part 820 | Drug cGMP under Parts 210/211; SiMD; SaMD |
| FDA AI guidance, docket FDA-2024-D-4689 | Draft, 7 January 2025, "not for implementation" | Evidence for submissions; seven-step credibility framework on a context of use | Manufacturing operations |
| EMA/FDA Guiding Principles | Non-binding, 14 January 2026, ten principles | Nonclinical, clinical, post-marketing and manufacturing | Not enforceable |
| ISPE GAMP 5 2nd Ed (July 2022), GAMP AI Guide (July 2025, 290pp) | Industry guidance | Method, lifecycle, practice | Not law — inspectors cite Annex 11, Annex 22, Part 211 |
| EU AI Act 2024/1689 | High-risk duties deferred to 2 December 2027 and 2 August 2028 | Annex I lists MDR and IVDR | Drug manufacturing — Directive 2001/83 absent, no Annex III use case |
Two errors dominate. CSA gets quoted as though it liberalised drug validation, when it is a device guidance under Part 820. The AI Act gets invoked as the forcing function, when it probably does not reach you at all — mechanics in a separate note.
Why must the model sit behind a deterministic decision boundary?
Because the draft makes determinism a scope condition rather than a design preference. Annex 22 applies to "models with a deterministic output which, when given identical inputs, provide identical outputs", and to static models only. Probabilistic and continuously-learning models "should not be used in critical GMP applications".
Only static, deterministic models are permitted in critical GMP applications under the draft. Determinism is not an architectural opinion. It is the scope paragraph.
Generative models are excluded from critical applications; where they are used in non-critical ones, the draft leaves a qualified human responsible for every output. "Annex 22 bans LLMs in pharma" is disprovable from the scope paragraph.
The boundary renders as an interface contract: §9.2 requires an appropriate threshold setting and asks you to consider flagging a low-confidence outcome "undecided" rather than guessing — a consideration in the draft, a hard state in any system worth qualifying. §9.1 puts the confidence score in the log for every prediction. §8.1 wants the features that drove a classification, naming SHAP, LIME and heat maps.
# pseudo-schema, not literal JSON
POST /inspect -> 200
{
"outcome": "PASS" | "FAIL" | "UNDECIDED", # Annex 22 9.2
"confidence": 0.0-1.0, # 9.1, persisted
"threshold": { "value": 0.94, "spec_id": "..." },
"attribution": [ { "feature": "...", "weight": ... } ], # 8.1
"model": { "id": "...", "digest": "sha256:...", "frozen_at": "..." },
"inputs_digest": "sha256:...",
"decision": null # null in every model response; a person writes this field elsewhere
}
Is the model's output a GMP record?
Yes, and the draft wants more of it recorded than most teams bother with. The output is a record. What it never becomes is the decision.
A model output cannot be attributable. There is no person behind a prediction, no role, and no stated reason for the change — which is why the signature is load-bearing rather than ceremonial.
ALCOA+ is a property of data and records; the audit trail is one control through which the property gets demonstrated. Authority is PIC/S PI 041-1, effective 1 July 2021, and MHRA's March 2018 guidance. Write it the other way round — "ALCOA+ applies to the audit trail" — and a data-integrity inspector knows within one sentence that you learned this from a slide.
Attributable is what a prediction structurally cannot be. Draft Annex 11 §12.2 wants user, role, old value, new value and an automatic prompt for the reason, "recorded at the time of events, not at the end of a process". §12.6 puts review with someone uninvolved, §12.8 before batch release, §12.10 in front of the QP at certification. So the record attributes the decision to whoever accepted the suggestion, and the system's job is to carry the inputs, the model digest and the accepting human's reason alongside it — that plumbing is an evidence layer.
What may a system suggest on a temperature excursion, and what must a human sign?
The model may rank, triage, predict, retrieve and propose. The alarm limit and the disposition belong to people, and the split has clause numbers behind it.
A model may score an excursion against stability data, pre-populate the deviation record and raise an early warning. Draft Annex 11 §8.2 requires alarm limits and early warnings to sit within approved, validated specifications, changeable only by privileged users under procedure. The model proposes a threshold; a change control approves it.
Acknowledgement is where the machine stops. §8.4 restricts acknowledgement of critical alarms to privileged users and demands a comment explaining why. §8.5 requires the alarm log to capture alarm name, both times, username, role and comment, uneditable by GMP users.
The US line predates AI and is sharper for it. 21 CFR 211.68(c) already lets automated equipment satisfy a required second check, one person verifying the equipment performed properly. Machine as verifier of an operation: permitted for decades. Machine as approver of a disposition: barred by §211.22(a).
What does the model have to beat, and who is allowed to test it?
The process it replaces, tested by people who have never touched the training side. Annex 22 §4.3 sets acceptance criteria "at least as high as the performance of the process it replaces", forcing you to have measured your humans first. Almost nobody has. §5.3 wants test labels verified by multiple experts or validated equipment; §5.6 says generating them with generative AI "is not recommended" and demands full justification for any use.
Then the requirement most ML teams cannot meet without a reorganisation. §6.2: employees involved in development and training must never have had access to the test data, and "there should be no copies of test data outside this repository". §6.3 wants a register of what was tested, when and how often. §6.5 demands controls stopping anyone who has seen the test data from training or validating the same model, with four-eyes pairing where separation is impossible. §6.4 extends independence to physical objects — the actual vials, the actual defects.
In a twelve-person data science team, test-data independence is an org chart change, not a config flag.
One catch that proves who read both documents: Annex 22 §4.3 cross-references "Annex 11 2.7" for the no-decrease principle, but in the Annex 11 consultation draft as published, that principle is §2.8. §2.7 is Security.
Does human-in-the-loop reduce your validation burden?
No. Leaning on a reviewer transfers the validation rather than removing it, and the trap sits in §3.3: where test effort has been reduced because a human checks the output, the intended use description must state the operator's responsibility, and "the training and consistent performance of the operator should be monitored like any other manual process". §10.5 adds that review records must be kept, and may imply "a consistent review and/or test of every output from the model".
Human-in-the-loop is not a discount on validation. It is a transfer: you stop qualifying the model and start qualifying the operator, with training records and a performance trend.
How do you version a model that has already been qualified?
Under change control and configuration control, across three things rather than one: the model, the system it runs in, and the process it automates. §10.1 covers all three before deployment and extends to "any change to physical objects the model is using as input", with any decision not to retest justified. The camera lens is in scope. So is the lighting rig and the tray geometry. §10.2 wants unauthorised change detectable rather than forbidden; §10.4 wants drift monitored against the qualified sample space.
"Version it like an SOP" undersells the obligation by half. Document control does not cover optics.
Where does pharma AI actually die?
At the moment somebody treats a model output as an approved document. On 2 April 2026 the FDA issued warning letter 722591 — the first AI-specific cGMP enforcement action, under a deficiency heading on inappropriate use of artificial intelligence. AI agents had generated drug product specifications, procedures and master production records; the firm used them without quality unit review and drew a citation under 21 CFR 211.22(c). Process validation was skipped too, and investigators were told the agent never flagged it.
Named failure modes, because unnamed risks never get budget:
- Oracle-as-approver. Model output enters a controlled document with no human approval act.
- Test-set osmosis. Whoever tuned the model curated the holdout. Accuracy looks excellent, means nothing.
- Confidence theatre. Score logged, never gated. No undecided state, so the model guesses at the boundary.
- Physical drift. New lens, new lighting, new supplier tray. Model unchanged, qualification void.
- Baseline by assertion. "Better than a human inspector", with no sample size, no inter-rater agreement.
- Machine-written audit trail. No user, no role, no reason. Fails Annex 11 §12.2 whatever the cryptography.
Can you rebuild the answer without calling the model? If not, you don't have a system. You have a demo with a try/except around it. The replay test is the question an inspector asks in different clothes. Rebuild determinism is a weekly gate in my own release process: same inputs, same digest, or the build does not ship.
What does a GxP AI validation protocol contain?
Eight groups, every line a mechanism rather than a restated obligation. Copy it, fork it, argue with it.
GxP AI VALIDATION — DECISION BOUNDARY PROTOCOL — v1.0 (22 July 2026)
The Hopium Lab. Supports a validation case. Does not confer one.
Clause refs are to DRAFT Annex 11 / Annex 22 (7 July 2025). Not legal advice.
INTENDED USE — before any test is executed
[ ] Intended use and input sample space documented and approved (22 §3.1)
[ ] Process SME named as owner, not the data science lead (22 §3.1)
[ ] Criticality decided: does the output touch patient safety, product
quality or data integrity? If yes, static and deterministic only
[ ] Operator responsibility stated where HITL reduces test effort (22 §3.3)
ACCEPTANCE CRITERIA
[ ] Performance of the CURRENT process measured, with method (22 §4.3)
[ ] Model criteria set at or above that measured baseline
[ ] No net increase in overall process risk, documented (11 §2.8)
TEST DATA AND INDEPENDENCE
[ ] Test repository access-controlled, audit-trailed, no copies (22 §6.2)
[ ] Register of which data was tested, when, how many times (22 §6.3)
[ ] Staff separation enforced, or four-eyes pairing justified (22 §6.5)
[ ] Physical test objects independent of training objects (22 §6.4)
[ ] Labels verified by multiple experts or validated method (22 §5.3)
[ ] No generative AI used for test data or labels, or use fully
justified (22 §5.6)
DECISION BOUNDARY
[ ] Three outcome states: accept / reject / undecided (22 §9.2)
[ ] Threshold is an approved specification, not a config default
[ ] No code path by which model output writes a controlled document
[ ] Alarm limits set within validated specifications (11 §8.2)
EXPLAINABILITY AND CONFIDENCE
[ ] Contributing features captured per decision (22 §8.1)
[ ] Confidence score persisted with every outcome (22 §9.1)
CHANGE AND CONFIGURATION CONTROL
[ ] Model, system AND process under change control (22 §10.1)
[ ] Physical inputs — optics, lighting, fixtures — in scope (22 §10.1)
[ ] Unauthorised change detectable, not merely prohibited (22 §10.2)
[ ] Any decision not to retest documented and justified (22 §10.1)
OPERATION
[ ] Input drift monitored against qualified sample space (22 §10.4)
[ ] Operator performance monitored like a manual process (22 §3.3)
[ ] Human review records retained per procedure (22 §10.5)
[ ] Test cases traceable to requirements (11 §9.5)
THE SIGNATURE
[ ] Audit trail captures user, role, old value, new value and reason,
written contemporaneously (11 §12.2)
[ ] Review by someone not involved in the activity (11 §12.6)
[ ] Review completed before batch release (11 §12.8)
[ ] Available to the QP at certification (11 §12.10); in the US, quality
unit approval under 21 CFR 211.22(a) and record review under 211.192
[ ] Signature breaks visibly if the record is later changed (11 §13.8)
The protocol does not wait on an adopted text. Static model, deterministic boundary, isolated test set, versioned optics, a threshold that is a specification, a signature with a reason attached — all buildable this quarter, all surviving whatever lands next. The draft is downstream of a constraint older than machine learning: a batch is released by a person who can be asked why.
Ross Jones, Founder, The Hopium Lab. Last modified 22 July 2026. Engineering guidance, not regulatory advice. Clause numbers cited are from consultation drafts and must be verified against adopted text.