Production-ready AI checklist: six things a demo never proves

"Production-ready" is the most abused phrase in AI automation. In most proposals it means the demo worked while someone was watching. A production-ready AI checklist should instead be a set of properties you can verify without being technical, and this is the one we hold our own six production systems to. Hand it to any vendor, us included, and ask them to confirm each item before handover. How they react to the list tells you almost as much as the answers.

1. Reliability

Demos run in calm conditions. Production runs through API outages, malformed emails and duplicate webhooks, and the difference shows up in four places.

• Retries. When an external service fails for a moment, and they all do, the workflow tries again instead of dropping the item. Ask what happens if the CRM is down for five minutes at 3am.
• Timeouts. A step that waits forever on a slow service blocks everything queued behind it.
• Dead-letter handling. An item that fails even after retries goes to a visible holding area for a person to look at. Ask where failed items go and who checks.
• Idempotency, a long word for a short promise: if the same trigger fires twice, the customer does not get two invoices. Ask what happens if this runs twice on the same input. Duplicate webhooks are common enough that we assume them by default.

2. Someone finds out when it breaks

An automation you cannot see is an automation that is quietly failing. The minimum is a log of every run (what came in, what went out, what failed), so that when a customer says they never got the confirmation, someone can find out why in minutes. Failures should post to a channel a named person reads: Slack, email, SMS, whichever that person opens. A dashboard nobody opens is decoration.

The cheapest early warning is a weekly run count. A workflow that normally handles 200 items and suddenly handles 12 is broken somewhere upstream, and volume is how you catch it before the customer does.

3. Quality

Anywhere a language model classifies, drafts or extracts, there should be a test set of real cases with known correct answers, and a score. "It usually gets it right" is a mood. "94 of 100 on your last three months of real emails" is a score.

The system should also have met your messy inputs before handover: the customer who writes in capitals, the invoice with a missing field, the PDF scanned sideways, the reply that quotes the entire thread. If it has only seen clean sample data, launch day is the test and you are the test environment.

4. A stranger could run it

The test is literal. Could a competent freelancer who has never met the builder read the documentation, understand what the system does, restart it, and change a setting? If everything lives in the builder's head, you have rented a person.

The settings you will want to change (email templates, price thresholds, recipient lists) should live somewhere editable, so that a small tweak does not become a billable engagement.

5. Ownership

API keys, the automation platform and the databases are registered to your company and billed to your card. Repositories and workflow exports live where you control them. At handover you receive every login and key, written up; the full version is the eight things you should hold at handover. If the vendor vanished tomorrow, everything keeps running and anyone you hire can pick it up. We have been the hired anyone more than once, and the difference between a rescue that takes a week and one that takes a month is usually whether this paragraph was true. When a builder disappears, the takeover starts with the accounts, and it only starts at all if you hold them.

6. After launch

• A named watcher. One specific person, yours or the vendor's under a care agreement that names monitoring and a fix window, whose job includes noticing when this system misbehaves. "We'll all keep an eye on it" means nobody will.
• A monthly report with numbers. Runs completed, failures, hours saved or items handled, cost to operate. Without it the system drifts into the large graveyard of automations that technically still run and practically do nothing.

Where the list does not apply

A script one person runs by hand on Friday afternoons does not need a dead-letter queue and a monthly report, and we would not quote for either. The list is for anything that runs unattended and touches customers or money. Between those two there is a judgement call, and our bias is toward over-instrumenting, because in the rescues we have scoped the item most often missing was monitoring.

How to use it

Send it with your request for proposal and ask which items are included at the quoted price, which cost extra, and which the vendor thinks unnecessary. All three answers are useful. A vendor who calls monitoring "overkill for a project this size" is telling you their systems fail silently. A vendor who ticks every box and cannot explain one in plain English ticked it for the sale.

Our fixed-price builds include the list, with a 30-day bug warranty after handover and every account and repository in the client's name from day one. If you have an existing automation that misses several of these, that is the situation our rescue service exists for.