An eval suite is just a test suite for behavior you can’t check with an equals sign. Ours has three parts: a golden dataset, a regression gate and a set of adversarial prompts.
The golden dataset
Start with 50 to 200 real cases, anonymized. Each one has the input, the behavior you expect and a sentence on why. Tag each case by difficulty and by the risk it covers. Every bug that reaches a user becomes a new case, so the set grows where the product is weakest.
Three kinds of checks
- Structured checks: valid JSON, required fields present, values inside allowed ranges. Cheap, exact, run on everything.
- Rubric checks: a short written rubric graded by a person, or by a model that is itself spot-checked by a person.
- Behavior checks: does it refuse when it should, cite the source it used, and say “I don’t know” instead of guessing?
An example case
{
"id": "pto-policy-017",
"tags": ["retrieval", "must-cite", "hard"],
"input": "Can I carry over unused PTO into next year?",
"context": ["handbook/leave.md#carry-over"],
"expect": {
"cites": "handbook/leave.md#carry-over",
"mentions": ["40 hours", "March 31"],
"must_not": ["guarantee", "legal advice"]
},
"why": "Users quote this answer to HR; a wrong cap causes disputes."
}Illustrative example. The policy details are invented for the sample.
The regression gate
The suite runs on every prompt change and every model upgrade. Results are compared with the last release, and a drop on any case tagged critical blocks the merge. Small drops elsewhere are allowed, but they’re visible in the pull request, so the trade-off is a decision rather than an accident.
Red-team prompts
- Instructions hidden inside retrieved documents (“ignore your rules and…”).
- Requests for another user’s data.
- Off-topic or harmful requests the feature should decline.
- Inputs in other languages, very long inputs and empty ones.
What changed for us
Prompt changes stopped being debates. A change ships when the suite says it’s no worse, and the suite gets one more case every time something slips through. That’s the whole trick, and it compounds.