Buying guide · · 4 min read · Lurto · Updated
Production-ready AI checklist: six things a demo never proves
Quick answer
Production-ready should mean a set of properties you can verify without being technical, not that the demo worked while someone was watching. There are six: reliability, meaning retries, timeouts, dead-letter handling and idempotency; someone finds out when it breaks; quality expressed as a score against real cases rather than a mood; documentation a stranger could operate from; every account and repository in your company's name; and after launch, a named watcher plus a monthly report with numbers in it. Send the list with your request for proposal and ask which items are included at the quoted price.
"Production-ready" is the most abused phrase in AI automation. In most proposals it means the demo worked while someone was watching. A production-ready AI checklist should instead be a set of properties you can verify without being technical, and this is the one we hold our own six production systems to. Hand it to any vendor, us included, and ask them to confirm each item before handover. How they react to the list tells you almost as much as the answers.
1. Reliability
Demos run in calm conditions. Production runs through API outages, malformed emails and duplicate webhooks, and the difference shows up in four places.
- Retries. When an external service fails for a moment, and they all do, the workflow tries again instead of dropping the item. Ask what happens if the CRM is down for five minutes at 3am.
- Timeouts. A step that waits forever on a slow service blocks everything queued behind it.
- Dead-letter handling. An item that fails even after retries goes to a visible holding area for a person to look at. Ask where failed items go and who checks.
- Idempotency, a long word for a short promise: if the same trigger fires twice, the customer does not get two invoices. Ask what happens if this runs twice on the same input. Duplicate webhooks are common enough that we assume them by default.
2. Someone finds out when it breaks
An automation you cannot see is an automation that is quietly failing. The minimum is a log of every run (what came in, what went out, what failed), so that when a customer says they never got the confirmation, someone can find out why in minutes. Failures should post to a channel a named person reads: Slack, email, SMS, whichever that person opens. A dashboard nobody opens is decoration.
The cheapest early warning is a weekly run count. A workflow that normally handles 200 items and suddenly handles 12 is broken somewhere upstream, and volume is how you catch it before the customer does.
3. Quality
Anywhere a language model classifies, drafts or extracts, there should be a test set of real cases with known correct answers, and a score. "It usually gets it right" is a mood. "94 of 100 on your last three months of real emails" is a score.
The system should also have met your messy inputs before handover: the customer who writes in capitals, the invoice with a missing field, the PDF scanned sideways, the reply that quotes the entire thread. If it has only seen clean sample data, launch day is the test and you are the test environment.
4. A stranger could run it
The test is literal. Could a competent freelancer who has never met the builder read the documentation, understand what the system does, restart it, and change a setting? If everything lives in the builder's head, you have rented a person.
The settings you will want to change (email templates, price thresholds, recipient lists) should live somewhere editable, so that a small tweak does not become a billable engagement.
5. Ownership
API keys, the automation platform and the databases are registered to your company and billed to your card. Repositories and workflow exports live where you control them. At handover you receive every login and key, written up; the full version is the ten things you should hold at handover. If the vendor vanished tomorrow, everything keeps running and anyone you hire can pick it up. We have been the hired anyone more than once, and the difference between a rescue that takes a week and one that takes a month is usually whether this paragraph was true. When a builder disappears, the takeover starts with the accounts, and it only starts at all if you hold them.
6. After launch
- A named watcher. One specific person, yours or the vendor's under a care agreement that names monitoring and a fix window, whose job includes noticing when this system misbehaves. "We'll all keep an eye on it" means nobody will.
- A monthly report with numbers. Runs completed, failures, hours saved or items handled, cost to operate. Without it the system drifts into the large graveyard of automations that technically still run and practically do nothing.
Where the list does not apply
A script one person runs by hand on Friday afternoons does not need a dead-letter queue and a monthly report, and we would not quote for either. The list is for anything that runs unattended and touches customers or money. Between those two there is a judgement call, and our bias is toward over-instrumenting, because in the rescues we have scoped the item most often missing was monitoring.
How to use it
Send it with your request for proposal and ask which items are included at the quoted price, which cost extra, and which the vendor thinks unnecessary. All three answers are useful. A vendor who calls monitoring "overkill for a project this size" is telling you their systems fail silently. A vendor who ticks every box and cannot explain one in plain English ticked it for the sale.
Our builds include the list, with a 30-day bug warranty after handover and every account and repository in the client's name from day one. If you have an existing automation that misses several of these, that is the situation our rescue service exists for.
Questions people ask about this
What questions prove a system is actually reliable?
Three, and they are answerable by a non-technical buyer. What happens if the CRM is down for five minutes at 3am? Where do items that fail even after retries go, and who checks that place? And what happens if this runs twice on the same input, because duplicate webhooks are common enough that we assume them by default, and idempotency is just the promise that the customer does not get two invoices.
What is the cheapest early warning that something has broken?
A weekly run count. A workflow that normally handles 200 items and suddenly handles 12 is broken somewhere upstream, and volume is how you catch that before the customer does. A dashboard nobody opens is decoration; failures have to post to a channel a named person actually reads.
How do we test whether the documentation is good enough?
The test is literal. Could a competent freelancer who has never met the builder read the documentation, understand what the system does, restart it and change a setting? If everything lives in the builder's head, you have rented a person rather than bought a system. The settings you will want to change, email templates, price thresholds, recipient lists, should also live somewhere editable, so a small tweak does not become a billable engagement.
Does every automation need all six items?
No. A script one person runs by hand on Friday afternoons does not need a dead-letter queue and a monthly report, and we would not quote for either. The list is for anything that runs unattended and touches customers or money. Between those two there is a judgement call, and our bias is toward over-instrumenting, because in the rescues we have scoped the item most often missing was monitoring.
What does a vendor's reaction to the checklist tell us?
Almost as much as their answers. A vendor who calls monitoring overkill for a project this size is telling you their systems fail silently. A vendor who ticks every box and cannot explain one of them in plain English ticked it for the sale.
More on buying guide
Have a system that needs this treatment?