In normal software, tests tell you whether a change broke something. AI features need the same discipline, but their outputs vary and "correct" is often a matter of degree. Evaluations (evals) are the tests for AI behaviour.
What goes into an eval set
- Representative inputs: real (anonymized) examples of what users or agents send.
- Expected outcomes or criteria: an exact answer, a passing unit test, required facts, forbidden content, a format.
- Scoring: automatic checks (tests pass, JSON valid, answer contains X), model-graded checks ("LLM as judge" with a rubric), and periodic human review.
- Non-functional measures: latency, token use and cost per request.
- Safety checks: off-topic or harmful requests, prompt injection attempts, data leakage.
When to run evals
On every change to a prompt, model, retrieval setup or tool — and on a schedule, because vendors update models. Treat a drop below threshold like a failing test: it blocks the release.
Offline and online
- Offline evals run before release against your fixed set.
- Online evaluation watches production: sampled human review, user feedback, canary releases and A/B tests.
A real example: GitHub Copilot
GitHub described in January 2025 how it evaluates models before offering them in Copilot. It runs more than 4,000 offline tests, using about 100 containerized repositories: for code changes it measures the percentage of unit tests passed and similarity to a known passing state; for chat it measures the percentage of questions answered correctly; it tracks token usage; and it tests safety. It then runs live internal evaluations, similar to canary testing, before wider rollout — and openly weighs trade-offs such as higher acceptance rates against higher latency.
Start small
Twenty to fifty well-chosen examples per use case beat none. Grow the set every time a bug or complaint reveals a gap.