Evals: The Skill That Separates a Real AI Product from a Demo

Evals are how we know if an AI agent is actually working — not just working once, in a demo, on the one input we tested. Traditional software has unit tests: same input, same output, pass or fail. AI agents don't work that way — the same prompt can produce a different answer tomorrow, and "correct" is often a judgment call, not a boolean. Evals are the discipline that replaces "it felt right when I tried it" with a repeatable, measurable answer to "is this pipeline good enough to ship, and where exactly is it breaking?"

This post is written for anyone on the team building or reviewing AI features — not just engineers. If you can read a spreadsheet, you can run an eval.

Why "just test it" doesn't work for AI

A traditional feature is deterministic: give it the same input twice, get the same output twice. An AI agent is not.

Traditional Software QA	AI Agent Evals
Same input → same output every time	Same input → output can vary run to run
Pass/fail is unambiguous	"Good enough" is often a judgment call
A test either matches or it doesn't	Output can be technically correct but still wrong for the user
Bugs show up as crashes or errors	Failures are silent — the agent just gives a confidently bad answer
Cheap to run (milliseconds, no inference cost)	Expensive to run (LLM calls cost time and money)

This is why "we tested it and it worked" means almost nothing for an AI feature. It worked once, on the examples you happened to try. Evals exist to answer the harder question: how often does it work, on the full range of inputs it will actually see in production?

A layman's example: the supplier risk alert agent

Let's ground this in something close to home. Imagine we build an AI agent whose job is:

"Read an incoming supplier news article, decide if it represents a real risk event (factory fire, bankruptcy, port closure, etc.), and if so, draft a one-paragraph alert for the customer explaining the impact."

This maps neatly onto our own SENSE → REASON → ACT → ESCALATE pattern: SENSE reads the article, REASON decides if it's a real event and how severe, ACT drafts the alert, ESCALATE routes it to a human if confidence is low.

Sounds simple. It isn't — because "read an article and decide if it's a real risk event" hides a mountain of unstated decisions, and that's exactly where evals earn their keep.

The three places an AI pipeline quietly breaks

Before you can measure anything, it helps to know where problems come from. In practice they cluster into three areas:

1. You don't actually know your own data. You can't read every incoming article by hand. At scale, you don't really know what the input looks like — some articles are one paragraph, some are 40; some are in English, some aren't; some mention the supplier once in passing and are actually about a competitor. You're flying blind on your own inputs before the model even runs.

2. Your instructions are vaguer than you think. "Decide if it's a real risk event and draft an alert" sounds precise. It isn't. Does a rumor count, or only confirmed events? Should the alert be two sentences or a paragraph? Should it name the specific product lines affected, or stay general? None of that is in the instruction — so the model is guessing, and every guess is a potential failure you'll get blamed for.

3. The model applies the rule to the wrong thing. An article about "Foxconn supplying screens for the new iPhone" might get flagged as a Foxconn risk event even though nothing bad happened — the model pattern-matched on the supplier's name appearing near dramatic language. This kind of failure — the model technically followed the instruction but generalized it incorrectly — never fully goes away, even with a great prompt. It just gets rarer.

Evals are how you find these three failure types systematically, instead of discovering them one angry customer email at a time.

The loop: Analyze → Measure → Improve

Once you accept that failures are inevitable and mostly invisible, the fix is a repeatable loop, not a one-time test pass.

Step	What you're doing	What it tells you
Analyze	Read a sample of real outputs on real inputs, by hand, looking for patterns	Surfaces failure types — e.g. "false positives on articles that just mention the supplier's name"
Measure	Turn each failure type into a specific, repeatable check you can run automatically	Tells you how often each failure happens, so you can prioritize
Improve	Fix the highest-impact failure — usually a prompt/instruction fix, sometimes a data or architecture fix	Closes the gap; you go back to Analyze to see what's next

The critical, easy-to-skip step is 'Analyze' — actually reading 20–50 real transcripts before writing a single automated check. Skipping straight to a dashboard of generic scores is the single most common mistake teams make, and it produces metrics nobody trusts and nobody acts on.

Two kinds of checks: reference-based vs. reference-free

Once you know what to check for, you need a way to check it. There are two flavors.

	Reference-based	Reference-free
How it works	Compare the output to a known correct answer	Judge the output on its own merits, no "correct answer" needed
Analogy	Grading against an answer key	Grading an essay against a rubric
Example for our agent	"Does the extracted supplier name exactly match the one in our system?"	"Does the alert avoid speculating about impact the article didn't actually state?"
Cost to build/maintain	Low — cheap, fast, deterministic	Higher — often needs a human or an "LLM-as-judge" to grade it
When to use	Whenever there's a clear right answer	Whenever quality is subjective, contextual, or has multiple valid forms

Rule of thumb: start cheap. A simple rule-based check (does the alert include a supplier name? is the JSON valid? is it under 100 words?) catches a surprising number of real problems for almost no cost. Save the expensive human- or LLM-graded checks for the failures that survive after you've fixed the obvious stuff.

How to actually score a subjective output

For anything that isn't a clean right/wrong — like "is this alert well-written and appropriately cautious?" — you need a way to turn a judgment call into a number. Three common approaches, roughly in order of how much effort they take:

• Direct grading — Look at one output, grade it against a rubric (usually Pass/Fail, not a 1–5 scale). This is the workhorse method and the one to start with.

• Pairwise comparison — Show two versions of the alert side by side, pick the better one. Useful when you're A/B testing two prompt versions.

• Ranking — Order three or more outputs best to worst. More informative, more effort — save it for when you're choosing among several candidate responses.

Why Pass/Fail beats a 1–5 scale: graders (human or AI) genuinely struggle to tell a 3 from a 4 consistently, and tend to cluster in the middle to avoid a hard call. "Did the alert overstate the risk — yes or no?" is a decision people can make quickly and consistently. If you want to track something more nuanced, break it into several separate binary checks (e.g., "names the affected product," "states severity," "avoids speculation") rather than one fuzzy score.

What this looks like for our supplier risk agent, end to end

1. Analyze: Pull 30 real alerts the agent generated last week. Read them. You notice a pattern — the agent flags articles that merely mention a supplier's name as risk events, even when the article is neutral news.

2. Measure: Write a binary check: "Does the source article actually describe a negative event (Y/N)?" Run it against 200 historical alerts. You find this happens 18% of the time — a real, sizeable problem, not a one-off.

3. Improve: Tighten the instruction — explicitly tell the model "only flag if the article describes a disruption, closure, bankruptcy, or safety incident; a supplier being mentioned is not sufficient." Re-run the same 200 examples.

4. Back to Analyze: Check whether the false-positive rate dropped — and look for the next most common failure pattern (maybe it's now under-flagging genuinely severe events).

That's the whole discipline. Nothing here requires a data science background — it requires reading real outputs, being specific about what "good" means, and being willing to count things instead of eyeballing them.

The one habit to build this week

If your team takes away one thing: before shipping any AI feature, someone has to sit down and read 20–30 real outputs, end to end, and write down what's actually wrong with them. Not a demo run. Not the three examples in the PRD. Real, messy, production-shaped inputs.

Everything else in evals — the metrics, the dashboards, the automated judges — is just a way to scale that first, unglamorous act of looking.