A real AI acceptance test, from start to finish
98 questions to an assistant on the Spanish low-voltage regulation, the criteria agreed in advance, the six serious errors and the signed report.
A real test, from start to finish. An assistant on the Spanish low-voltage regulation, tested on 2026-11-14. I built it for this demonstration with an open model running locally, so it is not a client’s system. Result: unfavourable. Four of the six criteria are not met, including serious errors, where the agreed threshold was zero.
Criteria:
- Errors in serious questions: required 0, measured 6, not met
- Correct answers: required ≥ 85%, measured 52/88, not met
- Says when the regulation does not cover it: required ≥ 90%, measured 9/10, met
- Corrects a false premise: required ≥ 75%, measured 2/8, not met
- Answers correctly in Catalan: required ≥ 80%, measured 3/6, not met
- Cites the right instruction when correct: required ≥ 80%, measured 51/52, met
The errors in serious questions:
- R-011. An appliance in zone 0 of a bathtub. It said IPX8. The regulation requires IPX7 (ITC-BT-27, table 1).
- R-018. Voltage limit on a construction site with a TT earthing system. It said 50 V. The correct value is 24 V AC (ITC-BT-33, 4.2).
- R-065. A socket 1 m from the bathtub and 1.2 m above the floor. The first time it did not answer; on the repeat it said zone 2. It is in zone 3 (ITC-BT-27).
- R-070. Extending an industrial unit from 60 to 130 kW. It was right that a design project is needed, but justified it with a 10 kW limit that does not exist. The rule is that the extension exceeds 50% (ITC-BT-04, 3.2.c).
- R-088. A 10 mm² neutral with 16 mm² phases in an indoor installation. It accepted it. The neutral must match the phases (ITC-BT-19, 2.2.2).
- R-097. Ingress protection of the bathroom light fitting. It did not give the IPX4 for zones 1 and 2 and mixed in swimming-pool requirements. It depends on the zone (ITC-BT-27, table 1).
Before changing the model, fix how the document's tables are read and add a premise check. Then repeat the test with the same cases.