Why “It Usually Works” Is Not Good Enough for AI
Many companies still assess AI applications like conventional software: a prototype produces impressive results, a small number of internal tests succeed and the system is released. This can be reasonable for deterministic software. It is insufficient for generative AI and AI agents.
A conventional system generally returns the same result for the same input. A large language model is probabilistic. Wording, context, model version, retrieved documents and configuration all influence the output. An AI agent is even more complex: it plans multiple steps, calls tools, gathers information, changes its intermediate state and may follow different paths to the same objective. A small early error can propagate through the entire workflow.
The management question is therefore not: “Is our model intelligent enough?” It is: “Can we prove that the complete system operates reliably, economically and within its rules under realistic conditions?”
Systematic AI evaluation provides this proof. It makes quality visible, creates release criteria and connects technical performance with business outcomes. For SMEs, this discipline is particularly important: limited budgets leave little room for prolonged trial and error, while one uncontrolled workflow can damage customer relationships, compliance or operations.
What AI Evaluation Really Means
AI evaluation, often called evals, is a repeatable process for measuring, comparing and improving AI systems. The evaluation target is not only the base model. It is the complete system of prompt, context, retrieval, data, tools, rules, interface and human oversight.
The NIST AI Risk Management Framework explicitly integrates trustworthiness into the design, development, use and evaluation of AI systems. Its approach combines governance, mapping, measurement and management and treats monitoring as a continuous responsibility rather than a one-time acceptance test.1 An AI application is never permanently “finished testing.” Models, data and usage patterns change and so does risk.
For AI agents, the object of evaluation expands further. Databricks describes agent evaluation as a lifecycle capability from experimentation and offline testing to production monitoring. Organizations must assess not only the outcome but also planning, retrieval, tool use, intermediate decisions and stability across repeated runs.2
The Four Quality Layers of an AI System
| Quality layer | Key question | Typical metrics |
|---|---|---|
| Output quality | Is the answer correct and useful? | Correctness, relevance, completeness, groundedness, tone |
| Process and trajectory quality | Did the system take an appropriate path? | Retrieval quality, tool selection, argument correctness, step count, policy adherence |
| Operational quality | Is the system stable, fast and economical? | Latency, error rate, repeatability, token use, cost per successful transaction |
| Business impact | Does AI improve a relevant business KPI? | Cycle time, conversion, first-contact resolution, error cost, escalation, adoption |
These layers are interdependent. A strong answer that takes too long or costs ten times the target is operationally inadequate. An agent that creates the right booking while accessing prohibited data is not production-ready. A chatbot with high response quality but low adoption creates no business value.
Why Public Benchmarks Are Insufficient
Model providers publish benchmarks for mathematics, coding, knowledge and reasoning. These are useful for broad comparison, but they do not show whether a system can reliably perform a company's specific process.
A model can lead on public benchmarks and still confuse internal product codes, recommend an unsuitable contract clause or apply the wrong escalation rule to a local-language customer request. Enterprise quality is contextual and must be measured with realistic cases from the organization’s own workflows.
Every production AI system therefore needs a quality contract: a documented definition of a good outcome, tolerable errors, errors that must never occur and the threshold at which a human must take over.
The Eval Operating System for SMEs
A pragmatic AI quality management system consists of seven building blocks. Together, they form an Eval Operating System that can be integrated into existing AI governance and operational reviews.
1. Define the Quality Contract and Risk Limits
Start with decisions, not tools. Business owners, domain experts, IT and, where relevant, privacy or compliance jointly define quality targets. A poor sentence from an internal marketing assistant is easy to correct. A system that processes payment data or generates legally relevant statements requires significantly tighter limits.
The quality contract should specify purpose, allowed and prohibited tasks, critical failure types, target metrics, minimum thresholds, escalation rules and ownership.
2. Build a Golden Dataset
The golden dataset is a curated collection of representative test cases. It includes common requests, difficult edge cases, known failure modes and, where possible, expected answers or evaluation rubrics. Fifty to one hundred high-quality cases are often enough to start. Coverage matters more than size.
| Test category | Purpose | Example |
|---|---|---|
| Normal cases | Measure everyday performance | Standard request from an existing customer |
| Edge cases | Test rare but plausible situations | Incomplete or contradictory information |
| Risk cases | Test rules and safety boundaries | Request for confidential customer data |
| Regression cases | Prevent known failures from returning | Case answered incorrectly by an earlier version |
The dataset grows from production. Relevant complaints, human corrections and escalations become new regression tests. In this way, operational experience becomes durable quality knowledge.
3. Run Offline Evals Before Every Change
Offline evals test a new prompt, model, knowledge base or agent logic against the golden dataset before real users are affected. The candidate is compared with the production baseline. A release proceeds only if minimum thresholds are met and no regression occurs in critical categories.
These release gates prevent a common AI failure: an improvement raises the average score while quietly degrading an important edge case. Without standardized comparison, quality becomes subjective.
4. Combine Deterministic Tests, LLM-as-a-Judge and Human Review
No evaluation method is sufficient alone.
Deterministic tests are appropriate for objective criteria: Was the correct tool called? Is a mandatory field present? Does the output match the schema? Was a prohibited source used? These tests are fast, inexpensive and reproducible.
LLM-as-a-Judge uses a language model to score open-ended outputs against a rubric, for example relevance, tone or completeness. It scales better than purely human review, but introduces its own bias and inconsistency. It therefore needs calibration against human judgments.
Human review remains essential for professional nuance, new error classes, ethical trade-offs and high-risk cases. The sensible model is risk-based: automated scoring at scale, human sampling for normal cases and mandatory approval for critical decisions.
5. Establish Production Tracing and Observability
Offline tests cannot represent every real situation. Production systems therefore need an auditable trace of request, prompt, model and version, retrieved sources, tool calls, arguments, intermediate errors, response, latency and cost.
This trajectory is especially important for agents. Recurring failure patterns include hallucinated tool calls, retry loops, invalid arguments, missing context, stale memory and unnecessary actions.2 Without tracing, the organization sees the final result but not the cause.
Observability must still respect privacy. Logs may contain personal or confidential information; retention, masking, access and deletion belong in the data-protection design.
6. Manage Quality, Cost and Business Impact Together
AI teams often optimize technical metrics while executives focus on productivity and outcomes. A shared scorecard closes the gap.
| Perspective | Core metrics | Management question |
|---|---|---|
| Quality | Correctness, groundedness, task completion | Does the system produce reliably useful results? |
| Risk | Policy violations, critical failures, unauthorized access | Does the system remain within its boundaries? |
| Operations | Latency, availability, repeatability | Does it perform under daily volume? |
| Economics | Cost per successful transaction, rework time | Is it less expensive than the alternative? |
| Business | Cycle time, revenue contribution, service quality | Does it improve the prioritized business KPI? |
The metric cost per successful transaction is particularly important. Cost per prompt or token can mislead. A cheap model becomes expensive when employees must repair every result. A more capable model may be more economical if it completes tasks correctly on the first attempt.
7. Embed a Continuous Improvement Cycle
AI quality can change without a code release. New documents alter retrieval, users ask different questions, data structures change and providers update models. NIST highlights the challenges of monitoring deployed AI systems, which makes continuous observation essential.3
An effective SME rhythm combines weekly operational review of exceptional cases, monthly scorecards for business owners and quarterly reassessment of risk, thresholds and model strategy. Critical incidents trigger an immediate root-cause analysis and new regression tests.