Evals for business automations: 30 real cases beat any spot check

Every AI automation we have shipped has broken at least once with nobody touching it. A model version changed under the prompt. A vendor adjusted a default. Someone edited one instruction to fix a complaint and quietly degraded ten other cases. If you run AI in production this will happen to you, and the only open question is whether a dashboard tells you or a customer does.

Evals for business automations are the answer we have settled on: a fixed set of real cases with expected outcomes, run by a script every time anything changes. Standard practice in ML research. Close to unknown in small-business automation, which is a big part of why so many of those automations rot.

What an eval set is

Thirty to a hundred real cases from your own operation, each paired with the outcome a competent employee would expect, stored in a file, and pushed through the live configuration by a script. That is the whole thing. It runs in minutes and you can compare this week's result against last week's.

For a phone assistant a case is one call scenario and the expected result: booked the right appointment type, captured the callback number, did not promise a discount that does not exist. For document extraction it is an invoice from your own books and the exact fields that should come out of it.

Why "it seemed fine" fails

Models update under you and the prompt was tuned against the old behavior. Every small prompt tweak that handles one complaint is an untested change to everything else the prompt does, and after a year of tweaks nobody can say what it guarantees any more. Meanwhile the business itself moves: new services, new prices, a policy that changed in March, and the system keeps answering confidently with last quarter's facts, which is how a chatbot starts making things up months after anyone last touched it.

Manual spot checks catch almost none of this, because people retry the cases they remember, and those are the cases that already worked.

Our own receptionist double-booked in week two

Our voice receptionist answers our business line, and it has an eval set that runs before every deploy. In its second week two calls hit the calendar within the same second and it booked both into one slot, which is one of the five ways a voice agent drops calls and double-books. Nothing in the prompt was wrong. The write to the calendar simply was not locked. That incident became a permanent test case, which is how the calendar-locking check got written, and it has run on every deploy since.

Most of the cases in our sets have that origin. We do not invent them. A failure happens once, it gets a case, and it never gets to happen quietly again.

Build one from what you already have

1. Pull the raw record: call transcripts, tickets, emails, submitted documents. The last 90 days is usually enough.
2. Sample for coverage. You want the common paths, the rare but expensive ones (cancellations, complaints, compliance questions), and every past incident that caused a fire drill.
3. Sit with the owner and write the expected outcome for each case. This step is where the value hides, because about half the time the business discovers it never decided what "correct" means for the edge cases, and the eval set forces that decision once, on paper, instead of letting the model improvise it on every call.
4. Script the run so it feeds every case through the production configuration and records the outputs. Runtime is minutes and the cost is usually a few dollars.

Gate the deploy

An eval set you run when you remember is a report. Wired into the deploy process it is insurance. Our rule on every system we operate: no prompt change, model upgrade or workflow edit reaches production until the run scores at or above the previous baseline. If the new version fixes the case you were targeting and drops three others, you see that first and choose the tradeoff on purpose.

Wiring this takes an afternoon, and it turns "we think the update is fine" into "the update passed all but two cases, and here are the two." An eval set is one line on the production-ready checklist, and the line most often missing.

Scoring conversational output

Structured outputs are easy: the extracted total matches the invoice or it does not. Conversations need more care, and this is where most homegrown attempts stall.

Use rubrics. Define four to six yes/no checks per case (answered the question asked, quoted the right price, offered the booking, invented no policy). A yes/no check can be graded consistently; a 1 to 10 "quality" score cannot. Then let a second model grade outputs against the rubric, which scales to hundreds of cases for a few dollars, and audit it: we read a random slice of judge verdicts every cycle, because an unaudited judge drifts like the system it grades. Disagreements between judge and rubric, or scores that moved since the last run, go to a person. They are a small fraction of the cases and nearly all the interesting failures live in them.

Zapier bought this discipline too

Zapier, which has an engineering organization most companies would envy, hired Fractional AI, an outside evals shop, to work on the AI side of its integration builder. The public case study puts the reduction in hallucinations above 80 percent. I read that as evidence that knowing how to build software and having the habit of measuring AI behavior are different skills; a company with hundreds of engineers still paid a specialist for the second one.

A five-person business running one AI receptionist and a couple of workflows needs it more, since it has no engineers to catch drift by accident. At that scale the whole discipline costs days.

Where an eval set is overkill: a workflow with no model in it, or a prompt that does one narrow thing with a structured output you can validate against a schema. Test that with the schema and a couple of fixtures and move on. Evals earn their cost once the output is language a customer will read or hear.

We build eval and reliability harnesses for existing AI systems as fixed-scope projects, usually $5,000 to $15,000 depending on how many systems and output types are involved, and every automation we build ships with one included. If you run AI in production and could not say today what its pass rate is, our build service page explains how we approach it.