An error, at least, announces itself. Sentry fires an alert, a chart turns red, and you sit down to fix it.
With a language model, the most common failure makes no noise.
I learned this on my own, with a side project I build to learn: a cycling coach where an LLM writes the weekly training plan. Then a set of rules in code adjusts it: don’t go over the time budget, don’t stack too many hard sessions, and so on.
#.The week that got easier
In August, a real plan came back with a strange peak week.
The peak week is supposed to be the hardest of the block. This one had 217 minutes. The build week before it had 327, and more load too.
Less time and less load than the week before. The block ended with a rest week nobody planned.
And no rule complained.
The model had been asked for 4 hours and 12 minutes and returned 3 hours and 37, 13.9% short. The tolerance was 15%. It stayed inside the margin by less than three minutes.
No error, no alert. Someone found it by reading the plan.
When I looked into why, I found something more uncomfortable. Every rule was a ceiling: cap the volume, cap the hard work. None of them put a floor under intensity.
The system was well protected against one kind of mistake and blind to the opposite one.
#.A test for answers that change
Anthropic puts it simply: an eval is a test for an AI system. You give it an input, grade the output, and measure whether it succeeded.
It sounds like a unit test. The difference is that a model can give you a different answer to the same input, and many questions have more than one good answer. An expect(result).toEqual(...) won’t tell you whether the answer is good.
My rules were guardrails: they check or fix each answer on the spot. Each one watched its own part. Nothing looked at the plan from the outside and asked whether it made sense as a plan.
That’s what an eval is for: the whole system, across many cases, over time. Of these hundred plans, how many peak weeks came out easier than the week before? And with the new prompt?
Would an eval have caught that week in advance? Honestly, probably not. I hadn’t imagined that failure, so I wouldn’t have written a check for it.
What an eval can do is keep it from coming back.
#.Read first, measure later
Hamel Husain and Shreya Shankar insist on something in their evals FAQ that’s easy to skip: before writing metrics, read real outputs by hand. Note what’s wrong, group the failures, and only then measure.
That’s exactly how it was found. Not on a dashboard. Reading.
And it applies to any product. A support bot that politely answers “please contact our team” to everything never breaks a rule. You only see it when you read the conversations.
Once you’ve seen a failure, checking for it is cheap. Mine took a few lines of code: the peak week can’t carry less load per minute than the week before. No other AI needed.
For what code can’t see, like whether an answer was clear, you can use another model as a judge. But compare its grades with your own first. A judge you haven’t checked is just one more opinion.
#.Start with your real failures
You don’t need thousands of cases. Anthropic says 20 to 50 simple tasks drawn from real failures is a great start. If you’re already in production, they point you to your bug tracker and support queue.
Every failure you read becomes a case. Every case protects the next change.
#.Why “almost always” isn’t enough
A plan is many answers in a row, and so is an AI agent’s task.
If each step had, independently, a 95% chance of coming out right, twelve in a row would come out clean only about 54% of the time.
That’s an illustration, not a measurement. But the point stands: the people using your product don’t experience your average success rate. They experience every single answer.
#.A change that doesn’t touch your code
The provider updates the model. Someone tweaks the prompt. A fallback kicks in and sends traffic to a different model when the main one is down.
Not a single line of your code changes, and your tests stay green. But your product may behave differently by morning.
Evals don’t remove that uncertainty. They turn it into something you can review on a Tuesday, before someone else finds it.