Evaluation is the moat, not the model
Everyone can call the same API. What separates a demo from a product is the test suite nobody sees.
Every team building on language models has access to roughly the same raw capability. The API key is not a competitive advantage. What actually differentiates the systems that survive their first thousand real users is far less exciting than model choice: a boring, well-maintained evaluation set.
The demo trap
A prototype gets built in an afternoon. It works on the six examples the engineer tried. It gets demoed, it lands well, and the project is declared feasible.
Then it meets production traffic, where the inputs are longer, messier, non-English half the time, occasionally adversarial, and frequently about edge cases nobody wrote down. The failure rate that felt like “roughly never” turns out to be four percent — and four percent of ten thousand daily requests is four hundred bad experiences a day.
The problem is not that the prototype was wrong. The problem is that nobody could tell you what its accuracy actually was, because there was no measurement.
What an evaluation set actually is
It is not a benchmark. Public benchmarks measure general capability; you need to measure your task.
A working evaluation set has four parts:
- Cases. Real inputs, drawn from real usage or realistic simulation, including the ugly ones. A hundred is enough to start; a thousand is comfortable.
- Expected behaviour. Not always an exact expected output — often a property: contains a citation, refuses politely, returns valid JSON matching the schema, does not invent a policy number.
- A scorer. Deterministic assertions where possible, a model-graded rubric where not. Model-graded scorers need their own spot check, because a grader that drifts is worse than no grader.
- A record. Every run, stored, with the model version, prompt version and score. This is the part teams skip, and it is the part that pays.
Why it pays
Three moments make the investment obvious.
The upgrade. A new model version ships. Is it better for your task, or just better on the vendor’s chart? Without evaluations, you find out from your users. With them, you find out in fifteen minutes.
The regression. Someone shortens a prompt to save tokens. Latency improves, cost drops, and a category of output silently degrades. An evaluation run catches this before merge; a human reading three examples does not.
The argument. When a stakeholder says the AI “feels worse this week”, you either have a number or you have a debate. The number ends the debate in either direction — including the times when the honest answer is that they are right.
Building it without stalling the project
The objection is always time. In practice, evaluation work compounds fastest when it starts small and early:
- Before writing the feature, write down twenty cases and what a good answer looks like. This alone surfaces disagreements about scope that would otherwise appear in week six.
- Automate scoring for the cases where scoring is mechanical — schema validity, required fields, refusal behaviour, latency, cost.
- Add every production failure to the set as it is found. A bug report is a free test case.
- Run the whole thing in CI. If it is not automatic, it will not be run.
None of this requires a platform. A folder of test cases, a script, and a stored results file will carry a team a long way.
The uncomfortable part
Sometimes the evaluation says the approach does not work. The retrieval cannot find the right passage often enough, the model cannot follow the constraint reliably, the task genuinely needs a human in the loop.
Learning that in week two is a good outcome. Learning it after launch is an incident. The evaluation set is what converts one into the other, which is why we build it first — and why we treat it as the deliverable that matters most on handover.
Articles reflect our engineering practice at the time of writing and are provided for general information only.