Testing an AI assistant in a hospital

Patient data, medical devices, the National Security Framework, Spanish and Catalan, and the time it takes your clinicians.


Testing an AI assistant in a hospital. Before an assistant answers your clinicians, I test it on your hospital’s own cases against criteria the hospital signs before work begins. The dated report records every error and how serious it is.

What can go wrong

An invented example of how it can fail: A theatre nurse asks for the antibiotic prophylaxis regimen for a hip arthroplasty. The assistant answers with the regimen from the 2019 internal protocol and cites it as its source. The protocol was revised in 2023 and the regimen changed. The answer cites a real hospital document, so nobody questions it, and if it reaches the patient it is a safety incident. Errors like this, serious and hard to spot, are the first the test looks for.

Questions the test would include:

Cases are prepared with pseudonymised data. Health data is a special category under Article 9 of the GDPR, and nothing leaves your environment without written authorisation. The test can run inside the hospital’s environment, with its controls and its users, so it rests on the security measures the hospital already applies, including those of Spain’s National Security Framework (ENS) where they apply.

The test gives you evidence of how the system behaves with your cases. I tell you how it fits into its conformity assessment under Regulation (EU) 2017/745 and into the work of its notified body.

The test can run in both languages with the same cases, and the report compares them. The acceptance test in both costs €6,900 plus VAT. Validating about 50 cases with their correct answer usually takes 6 to 10 hours of an experienced professional.

It is written for a quality, patient-safety or ethics committee to read without jargon, with the criteria, the results, examples of each failure and the limits of the test.

What the AI Act means here

An assistant that searches and cites internal documents, with no medical purpose of its own, is not in Annex III. If its intended purpose is medical, such as supporting diagnostic or treatment decisions, it may be a medical device under Regulation (EU) 2017/745. If its conformity assessment also requires a notified body, it is high-risk (Article 6(1) and Annex I), with requirements applying from 2 August 2028. Separately, Annex III, point 5(d), covers emergency healthcare patient triage systems from 2 December 2027.

In any case, if the assistant talks with people, they must be told they are talking to an AI unless that is obvious (Article 50, from 2 August 2026). And whoever develops it or uses it in their organisation must take measures to support the AI literacy of the staff who work with it (Article 4). Whether a system is high-risk is a legal question for your adviser. I do not classify it; the test gives you evidence either way.

A public hospital in Spain can order the test as a minor services contract (contrato menor): its €5,500 excluding VAT is below the €15,000 limit of Article 118 of Law 9/2017.