Why your voice agent drops calls and double-books, with the fixes

Voice is the least forgiving kind of AI automation. A chatbot that takes six seconds to answer is slow. A phone line that takes six seconds to answer is dead, and the caller has hung up. We run a voice receptionist on our own business line, we have listened to a lot of these calls, and every broken voice deployment we have scoped fails in one of five ways.

1. A voice agent drops calls when the silence gets too long

Callers say "hello? hello?" and hang up in the middle of the call. Your analytics show short calls with abrupt endings and no error anywhere.

The reason is arithmetic. A voice pipeline is a chain: speech-to-text hears the caller, a language model decides what to say, text-to-speech says it, and each stage adds delay. 800 ms of transcription plus 2 seconds of model time plus 700 ms of synthesis is 3.5 seconds of dead air on every turn. People tolerate about a second of silence on a phone before something feels off. Past two seconds they assume the line is gone.

Nothing in the system is failing, so nobody finds it in the logs.

Measure each stage on its own, because the fix depends on which one is fat. Then stream everything: transcription should emit words as they are spoken, the model should start its reply before the full answer is composed, and synthesis should begin speaking the first sentence while the rest is still generating. Use the fastest model that can do the job. Put a short acknowledgment ("sure, one moment") in front of anything slow like a calendar lookup. We aim for under 1.5 seconds from the caller finishing to the system starting to speak.

2. Barge-in

The caller interrupts to correct something and the system either plows through its sentence or stops mid-word every time someone coughs.

Letting the caller interrupt is a feature that has to be configured, and the defaults are wrong in both directions. Too insensitive and the system lectures over a person who is trying to say "no, I said Tuesday." Too sensitive and every background noise truncates the reply. The subtler bug: many builds stop the audio on interruption but still process the full turn as if it was delivered, so the system's idea of the conversation and what the caller heard drift apart.

Require speech, with a minimum duration, before treating a sound as an interruption. When one happens, the system needs to know how much of its sentence was played and update its own state to match. Then test it on purpose. Call in and interrupt rudely, repeatedly, mid-sentence, because that is what callers do.

3. Two calls, one slot, same second

Two customers turn up for the same appointment. It is rare, so everyone shrugs, until it happens again in the busiest week of the year.

Our own receptionist did this in its second week. Two calls hit the calendar within the same second, both saw the slot as free, and both booked it. The pattern underneath is check-then-book: the system confirms availability at the start of the booking flow, spends ninety seconds collecting a name and a phone number, then writes to the slot it checked earlier. Races like that surface exactly when volume is highest.

The fix we shipped is to verify the slot again at the moment of writing and treat "slot taken" as a normal conversational turn: "that time just went, I have 2:30 or 4:00 instead." If the calendar supports tentative holds, take one when the slot is offered and confirm or release it at the end. An availability answer older than a few seconds should be treated as a rumor.

4. Bookings that were confirmed and never existed

The caller was told "you're booked for Tuesday at 2." Nothing is on the calendar. You find out when they arrive.

The conversation and the booking write are separate systems glued with a webhook or an API call. When that call fails, times out, or returns an error nobody checks, the voice layer keeps going, because it never looked at the result. It confirms a booking it did not make. This is the most damaging of the five, since every occurrence is a person stood up in your lobby.

The confirmation the caller hears has to be generated from the booking system's response and never before it. On failure, retry; if it still fails, say so and take a callback number. Every failed write should alert a human within minutes, to a channel someone reads, with the caller's number and the requested slot so the booking can be finished by hand. An alert nobody reads is how an automation dies quietly in production.

5. Noise

Calls from cars, job sites, and kitchens go sideways: wrong names, wrong digits, the system answering things nobody said. Transcription accuracy drops sharply in noise and the model downstream trusts the transcript completely. Names and phone numbers suffer most, and those are the fields that matter.

Use a transcription model tuned for telephone audio, and design the conversation defensively. Read names and numbers back. Ask for spellings. When confidence is low, verify before proceeding. One confirmation turn costs less than one wrong callback number.

Score the transcript, never the impression

Do not judge a voice system by demoing it from a quiet office and feeling good about it. Place 20 real calls with different voices, accents, background noise, interruptions, and awkward requests, then score each one against the transcript and against what was written to the calendar. Kept as a fixed set and rerun before every deploy, those calls become an eval set for a business automation. We learned this on our own product: we once scored calls against our own transcription of the audio instead of the system's, shipped a change based on that, and had to revert it. Impressions lie, and so does a second-hand transcript.

One limitation. If your business gets a handful of calls a day, a voice agent may be more system than you need, and a good voicemail greeting with a same-day callback will serve those callers better than anything in this post. What an AI voice agent costs only starts making sense at roughly a hundred calls a month.

If the system is failing and you want the diagnosis done for you, we offer a written diagnostic for $500. We place and score the test calls, and within 3 business days you get a document naming your failure modes and a fixed quote, with the $500 credited against the work. Most voice fixes land between $1,500 and $4,000 and take 1 to 2 weeks, with a 30-day bug warranty. Start at our rescue service or tell us what your callers are hearing.