An AI assistant is intended to prepare quotations faster. A hundred cases were checked during the pilot and no errors were found. Is that enough to let the sales team use it? Let us treat this as an illustrative example, not a Fassbender client case.
The number sounds reassuring. But it does not yet tell us which errors the team looked for, who was able to detect them or whether the cases tested reflect the work ahead. An incorrect delivery commitment is not the same as a poorly worded paragraph. Classifying both simply as a “pass” does not provide a sound basis for a decision.
I would not base approval on “100 tests, zero errors”, but on the scope of use supported by the evidence and the limits of that evidence. This does more than guard against poor decisions. It also prevents a team from unnecessarily holding back a useful, limited application because it is being asked to demonstrate absolute safety.
What a hundred error-free cases actually establish
An explicitly illustrative calculation shows the limit. Suppose a hundred independent cases have been selected and assessed against a clear, binary definition of an error, with the system unchanged throughout. Each case has the same probability of error in the statistical model. For the result to apply to operations, the selection must also be representative of the relevant cases in actual use.
With zero observed errors, the one-sided exact 95% upper confidence limit for the error rate is approximately 3%. This is based on the binomial distribution; the calculation gives approximately 2.95%. NIST describes the method for exact confidence limits. This is not a measured AI error rate, nor is it the expected error rate of your system. The 95% refers to the confidence level of the procedure, not the probability that your pilot is safe. 1
For management, the first implication is straightforward: zero observed errors are not evidence of zero risk. Even this limited statistical statement holds only if the assumptions are appropriate. A hundred variations of the same simple request are no substitute for a hundred independent cases drawn from the intended use.
More tests can strengthen the evidence. They cannot, however, compensate for an inappropriate definition of an error or an important type of case that was never tested.
Describe the evidence before discussing approval
Before the next pilot review, I would ask for a brief test record. The person responsible for the work in question should endorse it, rather than leaving it solely to the development team. One page should cover:
- ◆Scope: What specific work was tested and with which model, source and configuration versions?
- ◆Definition of an error: What counts as substantively incorrect or not permitted and who assessed the results?
- ◆Selection: Where did the cases come from, which situations in use do they cover and what is missing?
- ◆Limits: Which use does the evidence support and which is explicitly excluded?
This is not another layer of reporting bureaucracy. Without these details, management cannot meaningfully assess the report of success. The team will also struggle to reproduce it later.
In the quotation example, the pilot might cover only the assembly of information that has already been approved. Making price changes independently would then not have been tested at all. Error-free assembly of information does not justify extending the scope to pricing decisions.
It must also remain clear what evidence exists for frequent standard cases and for rare exceptions with serious consequences. An impermissible commitment must not disappear within a favourable overall average. How much residual uncertainty is acceptable depends on its consequences and the controls in place. That is a management decision, not a requirement set by a generic benchmark.
The voluntary US reference framework NIST AI 600-1, published in 2024, recommends testing AI behaviour in realistic scenarios. It warns of gaps in measurement between controlled tests and actual use. It provides guidance, not certification or a legal guarantee. 2
Limited evidence calls for limited use
If the test record supports a useful, clearly defined application, I would limit its use accordingly. The assistant might, for example, prepare information for a draft quotation. A person with the relevant expertise and authority still checks the binding terms before the quotation is sent out.
The control must be practicable. A mandatory review is of little help if no one can see the necessary sources or has time to check them. The additional review effort therefore belongs in the assessment of economic viability. What matters is the time taken to produce a quotation that has been checked and approved, not just the speed of the first draft.
Some cases should be left untouched until the final assessment. Otherwise, the test may show only how well the team has improved performance on tasks it already knows. When sources, model versions or the scope of use change, it is necessary to reassess which evidence remains valid.
If such limits on use and approval criteria still need to be decided across several workstreams, the Executive AI Transformation Blueprint can establish the management decision architecture for doing so. The three-week sprint sets priorities, assigns responsibility and defines how value will be evidenced. It does not replace subsequent testing in operations.
At the next review, I would therefore ask to see more than the hundred green results. I would ask: what specific approval do these tests support and what do they not support?