Definition: what is AI model evaluation?
AI model evaluation is a structured process for checking whether an AI system performs adequately for a specified purpose. It examines outputs, errors, limitations and risks using defined tasks and evidence. Evaluation is not simply asking whether an answer sounds helpful, and it is not identical to choosing the model with the highest public benchmark score.
For a workplace team, the object being evaluated is usually the entire workflow: the model, instructions, documents, connected tools and human review. A model may perform well in a general test but fail on an organisation's policy documents or terminology. The same model can also behave differently when prompts, source material or access permissions change.
A source-backed framework
The NIST AI Risk Management Framework organises risk management around four functions: Govern, Map, Measure and Manage. Measurement belongs inside that wider process. Teams first need to understand the purpose and context, then evaluate relevant characteristics and decide what to do with the results.
NIST's Generative AI Profile, AI 600-1, addresses risks specific to generative systems, including confabulation and information integrity. These references support risk-aware testing; they do not prescribe one accuracy threshold for all organisations. A suitable threshold depends on the task, affected people and consequences of failure.
This article contains no vendor performance claims. All numerical values in its worked example are hypothetical and provided to explain calculations. They should not be quoted as research findings, promised productivity improvements or evidence that a real system is approved.
Why a polished answer is not enough
A language model can produce a confident answer that misstates a policy, omits an exception or cites a source that does not support the claim. Fluency can make such errors harder to notice. Evaluation therefore separates presentation quality from factual correctness and from operational safety.
For example, an onboarding assistant may produce an attractive checklist while inventing a probation rule. A recruiting assistant may follow the requested format while adding an unsupported candidate judgment. A meeting summariser may capture the main discussion but invent an action owner. Each failure needs a different test rather than a generic helpfulness rating.
Define what good means before testing
Start with the business task and write observable requirements. An approved policy summary might need to preserve eligibility conditions, cite the correct source, avoid personal data and say when evidence is missing. Decide which errors are tolerable with correction and which require stopping the workflow immediately.
Use reference evidence that people can inspect. For document questions, that may be a current approved policy. For a classification task, it may be an independently reviewed set of labels. Where reasonable reviewers disagree, record the disagreement instead of pretending there is a perfectly objective answer. A test becomes stronger when its assumptions are explicit.
Example: evaluating an HR policy assistant
Imagine a fictional HR team building a read-only assistant that answers questions from an approved handbook. The team creates 40 synthetic questions: 20 ordinary policy questions, 10 questions involving exceptions and 10 questions the handbook cannot answer. Those are illustrative design choices, not a recommended minimum sample size.
Reviewers assess factual support, completeness, citation quality and appropriate uncertainty. Suppose the assistant gives fully supported answers to 24 of the 30 answerable questions. Its supported-answer rate for that subset is 24 divided by 30, or 80%. Reporting 24 divided by all 40 questions would answer a different question and obscure how unanswerable cases were handled.
Suppose it correctly states that evidence is missing on 8 of the 10 unanswerable questions. Its appropriate-abstention rate on that subset is 80%. The team reports these results separately. Combining them into a single success score could hide a dangerous tendency to invent answers when policy evidence is absent.
The reviewers inspect the failures and find that several involve an outdated handbook section. They correct the source process, then test on new questions. They do not claim the assistant improved solely because it now answers the original test correctly. The system remains a draft-support tool until the agreed release criteria are met.
Metrics to choose from
| Metric or check | Suitable question | Limitation to remember |
|---|---|---|
| Supported-answer rate | Are factual claims supported by authorised evidence? | Requires careful source review |
| Completeness | Are required conditions and exceptions included? | Needs a task-specific rubric |
| Citation correctness | Does the cited passage support the claim? | A real link can still be irrelevant |
| Appropriate abstention | Does the system admit missing evidence? | Excessive refusal can make it unusable |
| Safety and privacy checks | Does the workflow respect data and action boundaries? | Passing a finite test is not a guarantee |
| Reviewer correction effort | What work remains before an output is usable? | Depends on reviewer expertise and task difficulty |
| Latency and cost | Is the workflow practical to operate? | Cheap, fast failures are still failures |
For ranking or classification, precision, recall and error rates may be relevant. For open-ended writing, rubric-based review is often more useful than exact text matching. Choose metrics because they reflect the job, not because they are easy to display in a dashboard.
How to evaluate an AI workflow step by step
Step 1: write a use-case specification
Identify users, inputs, outputs and permitted actions. Define the source of truth and the consequences of a wrong answer. Include a list of excluded uses. An HR assistant that summarises policies must not quietly become a system that decides employee eligibility or makes employment recommendations.
Step 2: build a test set with realistic variation
Include common cases, difficult cases and missing-information cases. Vary wording, document length and relevant exceptions. Use synthetic or appropriately authorised data. Do not expose personal records just to make a test feel realistic. Keep a separate set of cases for checking revisions so the team does not optimise only for familiar examples.
Step 3: create a clear review rubric
Describe each rating in observable terms. For factual support, distinguish correct and supported, plausible but unsupported, and contradicted by the source. Specify how omissions and privacy failures are handled. Ask reviewers to cite the evidence behind their rating. A rubric with vague labels such as good or bad will produce inconsistent results.
Step 4: capture the conditions of the run
Record the model or product version when available, the prompt, source versions and relevant settings. Preserve the output and review result. If the vendor does not expose a model identifier, record what information is available rather than invent one. Results need enough context to be interpreted and repeated meaningfully.
Step 5: use independent human review
Review important outputs against the evidence, not against another generated answer. Where feasible, compare judgments from more than one qualified reviewer and resolve disagreements. An AI judge can help organise a large review workload, but it may share blind spots or favour convincing prose. Validate its judgments before relying on them.
Step 6: analyse failure categories
Separate unsupported facts, missed exceptions, wrong citations, unnecessary refusals, privacy problems and unauthorised actions. Aggregate scores are useful for tracking, but categories identify remedies. A retrieval failure may need better source handling; an ambiguous task may need a clearer specification; a dangerous permission may need to be removed entirely.
Step 7: define release and stop criteria
Agree on acceptable performance for the task and explicit conditions that block release. Do this before seeing the preferred system's results. For consequential workflows, involve appropriate domain, privacy and security specialists. Do not copy an arbitrary pass rate from a course quiz or another organisation and treat it as a safety standard.
Step 8: retest and monitor
Evaluate revisions on fresh cases and keep earlier failure cases as regression checks. After deployment, collect appropriate feedback and review incidents. Repeat relevant tests when the model, sources, prompt or permissions change. Evaluation is an ongoing operating practice, not a certificate obtained once at launch.
How to report results honestly
State the task, sample composition, evaluation date, review method and limitations. Show counts alongside percentages so readers understand the denominator. Distinguish a pilot result from production performance. Explain whether the sample was representative or intentionally concentrated on difficult cases. Avoid claiming a small test proves universal reliability.
If a workflow reduces editing effort, describe how that effort was measured and whether quality remained acceptable. A faster first draft is not automatically a better final result. Where evidence is incomplete, write that the result is uncertain. This is more useful for decisions than an impressive number without a defensible method.
Common mistakes
Using the model to create the test answers and grade itself can conceal shared errors. Evaluating only successful demonstrations excludes the situations users actually struggle with. Changing the prompt and the test simultaneously makes comparison difficult. Reporting a percentage without its denominator makes even a correct calculation misleading.
Another mistake is testing only the model while ignoring tools and reviewers. A system can answer correctly but disclose information to the wrong person. A reviewer can approve a flawed output because the interface hides evidence. Evaluate the workflow that real employees will use, including access boundaries and final sign-off.
Frequently asked questions
Is evaluation the same as a public benchmark?
No. A benchmark is a standardised test useful for certain comparisons. Workplace evaluation checks suitability for a particular task, data environment and risk profile. Public results can inform selection, but they do not replace local testing.
How many examples should we test?
There is no universal number. The required evidence depends on variation, expected error frequency and consequences. A small pilot can reveal obvious problems but cannot establish rare-event safety. Seek statistical and domain expertise when consequential decisions require stronger evidence.
Can HR evaluate AI without writing code?
Yes. A structured set of synthetic cases, an approved reference document and a spreadsheet rubric can support an initial review. Technical colleagues are still needed for access controls, logging and connected actions. Start with a narrow task and avoid claiming more than the evidence supports.
Key takeaways and sources
Evaluate the full workflow against explicit requirements. Use representative evidence, separate error categories and keep denominators visible. Treat unsupported answers and privacy failures differently from cosmetic issues. Retest after meaningful changes and monitor how people use the system.
Primary references: NIST AI RMF and NIST Generative AI Profile. Continue with AI hallucination, AI bias and prompt injection.