10 questions before hiring an AI agency, with the bad answers

The AI automation market has almost no barrier to entry. Anyone with a no-code tool and a landing page can call themselves an agency, and plenty do. The useful part is that a handful of direct questions before hiring an AI agency will separate the people who run production systems from the people who run demos, usually inside one call. Here are the ten we would ask, with the answers that should worry you.

1. Who owns the accounts and the code?

Everything should be built inside your accounts: your OpenAI key, your automation platform, your database, your repositories. If the agency holds the accounts, the agency holds you, and firing them means starting from zero. Three of the six rescues we have scoped began the same way, with the original builder going quiet and the client discovering that every login belonged to someone who no longer answered email. Ask to see their handover checklist of accounts, keys and exports; a vendor who runs one will send it without editing it first.

The answer to worry about is "we manage the infrastructure so you don't have to think about it." That sentence describes a welded exit door.

2. What happens when it breaks at 2am?

Automations fail. APIs change, tokens expire, models get swapped under you. The question is who notices, how fast, and what response time is in the contract. "Email us and we'll take a look" means there is no service level, and a service level that exists only in conversation does not exist.

3. Price before the call

Published pricing means the agency has defined its scope tightly enough to commit to numbers in public, which is why we publish ours and what 20 agency price lists showed when we compared them. Hidden pricing means the number will depend on how your company looks on LinkedIn. In our view this is the strongest filter on the list, because it is the one that cannot be faked in a meeting. "Every project is different, let's hop on a call" is the standard reply. Some projects are different. The starting range never is.

4. What is in scope, and what counts as a change?

Vague scope is the main source of blown budgets. You want a written list of what is included and a written process for changes: what triggers a change order, how it is priced, who signs off. "We work agile, scope evolves as we learn" sounds reasonable and means the bill evolves too.

5. Evals or vibes

Any workflow built on a language model needs an evaluation set scored before every deploy: real inputs from your business, expected outputs, a score before handover. Ask to see one from a past project, redacted if necessary. An agency that tests by trying a few prompts until the output looks fine is shipping vibes. A blank look at the word "eval" ends this conversation for us, as does "the models are very reliable now."

6. Monthly running cost of the last project

Build cost is half the picture. API usage, platform subscriptions and upkeep are the other half, and some cheap builds are expensive to operate. Ask about the four meters behind a monthly automation bill by name. An agency should know this number for its last three clients, at least roughly. "It depends on usage" is true and is also a way of declining to answer.

7. Is there a bug warranty?

A defined period after handover, written into the contract, where defects are fixed free. Ours is 30 days. Without one, every bug found after launch is a new invoice, which quietly rewards sloppy delivery. "We stand behind our work" is a sentiment. Ask them to stand behind it in clause form.

8. Offboarding

What happens if you stop working together? The good answer is specific and boring: credentials transferred, documentation handed over, systems keep running. If the answer involves things breaking or a buyout fee, you are being sold dependency.

9. Can I talk to a client from a year ago?

Twelve months, specifically. Anyone can produce a happy client two weeks after launch. A reference at a year means the system survived model updates, staff turnover and ordinary neglect. "Our client work is confidential" is the usual dodge, and one reference breaches nothing.

A fair caveat: an agency younger than a year cannot pass this test, and some good ones are that young. Ask for the oldest reference they have and weigh it accordingly.

10. What do you turn down?

Competent specialists have edges. An agency that says yes to chatbots, custom models, mobile apps, SEO and web design is a general dev shop with an AI landing page. Ask what they refuse and where they send people instead. "We handle the full spectrum of AI" is the answer to walk away from.

Red flags

Guaranteed ROI percentages. "Our clients see 300% ROI" is unverifiable and usually invented; a serious firm shows the arithmetic for one named case instead.

Pressure to sign a twelve-month retainer before anything has shipped. Retainers are legitimate once a build has proved itself. Before that they are a way to get paid regardless of outcome.

A portfolio made of screen recordings belongs here too, though it is a softer signal. Prototypes are easy to film. Ask how many of them are running inside a paying customer's business today and for how long. The industry-wide answer is often "a few," which is exactly why the question is worth asking.

Print the ten and ask them in order. A good agency will enjoy the conversation, because the questions filter out its worst competitors. How we answer them, including scope, warranty and ownership terms, is on the pricing page before any call.