How an AI acceptance test is done
How an AI assistant is accepted with the method of industry: a failure analysis first, and criteria signed before looking.
How a test is done. The order in which industry accepts a machine, applied to an AI assistant: first how it can fail is analysed, and only then is it tested against criteria already signed.
FMEA: what can fail and how much it matters
I start with a failure mode and effects analysis (FMEA) of your assistant, as with a machine before it goes into service: what can go wrong and how much each failure would matter. The serious questions come from it, and they are the ones that count first.
Above all it looks for the failures that pass for right, such as a wrong figure or a document no longer in force, stated with full confidence.
As in an industrial FMEA, each failure mode is rated by its severity and its likelihood, and weighs more if it would be hard to catch before it does harm. The most serious get the most questions.
In the diagnosis, this analysis is the two-page document with your risks ranked by severity.
Criteria signed before looking
With the analysis in front of you, the acceptance criteria and their thresholds are agreed, for example no serious error and at least 85% correct answers. You sign them before a single question is run.
The criteria file is sealed with its SHA-256 fingerprint, which is recorded in the test record. If a threshold were changed after seeing the results, the fingerprint would no longer match.
The criteria of the demonstration test were set on 2026-11-14, before it was run. This is their fingerprint:
Anyone with the file can recompute it and check that nothing has changed. In an engagement, the set of questions also carries its fingerprint in the signed protocol.
FAT and SAT testing
In industry, equipment is tested twice before it is accepted: on the maker’s bench, the FAT (factory acceptance test), and at the client’s plant, the SAT (site acceptance test). An AI test follows the same logic.
The FAT is the test on the test bench, on a version that does not change while it lasts, with the serious questions run twice. The SAT runs the same questions again in your real environment, with the test bench that stays in your organisation.
Serious questions are run twice because an assistant does not always answer the same way: in the demonstration test, 2 of the 52 changed their result when repeated.
The acceptance test includes the full FAT, with up to 120 questions, and a repeat of up to ten questions after the fix.
It does not include an on-site SAT. The test bench and the instructions to repeat the test stay in your organisation, and your team runs the SAT on the system in service and after every important change.
If you prefer, the whole test can run inside your environment from the start: the FAT then takes place where the assistant will work.
The report and its three results
The report compares what was measured with what you signed and sets out every failure with its proof. I sign and date it, and it holds for the version and the cases tested.
- Favourable: Every signed criterion is met.
- Unfavourable: At least one criterion is not met. The report says which, and what to change.
- Inconclusive: What was measured is not enough to decide under the signed rules, for example if the version changed during the test.
Your committee or your legal team can use it as documented evidence that the system was tested. It does not replace the conformity assessment under the AI Act, nor a notified body where one is required. The method takes the technical specification ISO/IEC TS 42119-2:2025 as its reference, which applies the ISO/IEC/IEEE 29119 software testing standards to AI. Using it as a reference implies neither conformity nor third-party certification.