Not in critical uses, if the draft holds. The EU’s draft GMP annex on artificial intelligence, Annex 22, says generative AI and large language models “should not be used in critical GMP applications,” and makes qualified staff responsible for their output in non-critical work. The draft is not final, and a report the European Medicines Agency (EMA) published on October 2, 2026 records industry asking regulators to judge each use by its purpose and its controls instead.

Annex 22 is written for drug makers. One question from EMA’s workshop applies well beyond them: what does the person who reviews an AI’s output need to know to challenge it?

What Annex 22 is, and whether it is in force

Good manufacturing practice (GMP) is the set of rules and guidance on how regulated products are made and controlled. For medicines in the EU, the detailed guidance sits in EudraLex Volume 4, the EU GMP guide, as chapters and numbered annexes. Annex 11 covers computerized systems. Annex 22, “Artificial Intelligence,” is planned as a new annex “intended to provide additional guidance to Annex 11 for computerised systems in which AI models are embedded,” in the words of EMA’s report.

The draft went out for public consultation from July 7 to October 7, 2025, alongside revised drafts of Chapter 4 (Documentation) and Annex 11. It is still a draft. As of October 28, 2026, the European Commission’s EudraLex Volume 4 page lists no Annex 22 and still lists Annex 11 at its January 2011 revision. EMA’s report says the workshop’s outcomes “will be used to inform the ongoing revision of Annex 22,” and gives no date for a final text.

What the draft says about generative AI

The draft applies to AI models used in the manufacture of medicines and active substances in critical applications, those “with direct impact on patient safety, product quality or data integrity.” Within that scope it covers two kinds of model only:

• Static models, which do not adapt their performance during use. Models that “continuously and automatically learn and adapt performance during use” are outside the annex and “should not be used in critical GMP applications.”
• Deterministic models, which give identical outputs for identical inputs. Models whose output might differ for the same input are excluded on the same terms.

The draft then excludes generative AI, which produces text or other content, and large language models (LLMs), the kind of model behind chat assistants: “Following the above, the document does not apply to Generative AI and Large Language Models (LLM), and such models should not be used in critical GMP applications.” Where they are used in non-critical work, “personnel with adequate qualification and training should always be responsible for ensuring that the outputs from such models are suitable for the intended use,” an arrangement the draft calls a human-in-the-loop (HITL).

Why EMA held an expert workshop on Annex 22

Comments on the draft went the other way. EMA’s report says they “indicated support for allowing manufacturers to use Large Language Models and generative AI systems in GMP applications, whether critical or non-critical.” The drafting group then discussed allowing such models where they “are supported by a fully documented, robust, risk-based control strategy.” But “concerns remain about the elements needed for such a control strategy,” so it asked to hear from AI experts.

The workshop ran online on June 30 and July 1, 2026. Industry associations registered with EMA sent a consolidated answer to the drafting group’s questions across six topics, from validation to human oversight. EMA published the report on October 2, 2026.

The report records a discussion, not a decision. Its conclusion says the discussions “suggest that subject to appropriate controls and a robust risk-based framework, there may be viable pathways” for adaptive, probabilistic and generative AI in GMP-regulated applications.

What industry asked for: judge the use, not the technology

The industry position was that these models “should not be excluded from GMP applications on the basis of a technology label alone.” The test instead should be “whether the model can be demonstrated to be fit for its intended use and whether the associated risks can be controlled within an existing pharmaceutical quality system.”

It was not a request for a free pass. Asked about hallucination, an AI producing fluent output that is false, the response was that if a manufacturer “cannot demonstrate fitness for intended use” in view of such failure modes, “the use cannot be justified.” One key message reads: “A valid QRM outcome may still be ‘do not use’ where uncertainty cannot be controlled,” QRM being quality risk management. Another: “Accountability remains with qualified/responsible persons and cannot be delegated to AI.”

What a human reviewer of AI output needs to know

The industry presentation on oversight separated two ideas. Human oversight “asks whether a qualified, accountable human can understand, challenge, intervene, override, reject or escalate.” Human-in-the-loop is narrower: the process “waits for a human action before a GMP-relevant workflow proceeds.” The report’s key message: “Human-in-the-loop is one possible control, not a default requirement for all AI uses.”

Regulators pressed on whether oversight can be meaningful in practice. They asked how to monitor the person doing it, how to manage automation bias (the tendency to accept a machine’s output without enough scrutiny), what information and competence a reviewer needs, and whether a human in the loop becomes a bottleneck. The answer on record sets the bar:

As potential evidence of oversight, the responses named training, qualification, knowledge testing, audit trails, records of AI outputs and of the changes people made to them, and monitoring of how person and AI perform together. One key message closes a gap a review step can hide: oversight “cannot compensate for inadequate validation.”

In the US, FDA’s April 2026 warning letter to a drug maker that had used AI to write its specifications and procedures turned on the same person, the reviewer, and a separate article covers that letter.

What this means outside drug manufacturing

Annex 22 is written for the manufacture of medicines and active substances. A cosmetics, supplement or food company is not its audience unless it also makes medicines. The reviewer question still applies to any quality team that uses AI, or any other tool, to draft or check a document: could the person who signed off have challenged what they signed?

A practical way to apply that test to your own AI use:

• Write down the intended use of each AI step: what it reads, what it produces, and which decision its output feeds.
• Sort the uses by impact. An error that could reach product quality, consumer safety or the integrity of a record needs more control than one that costs a redraft.
• Name the reviewer for each use and check what they know: the process, the output, the decision it feeds, and where the tool tends to go wrong.
• Give the reviewer the reference the output must meet, such as the approved specification, so an omission shows. With only the output in hand, a reviewer can check what is there, not what is missing.
• Keep what the AI produced next to what the reviewer changed, and who approved it.
• Record the uses you decided against, and why. “Do not use” is a valid outcome.
• When you cite Annex 22, cite it as a draft and date its status to the source you read.

A written reason gives a reviewer something to challenge. Smart-Suited Tech’s document agent, Elara, resolves ordinary conflicts between documents by rule, with the reason on the record, and surfaces compliance-critical changes for a person to decide.