People forgive a lot from software. They don't forgive silence on a phone call. A typing indicator on a chat app is charming; a three-second pause on a live call is a dealbreaker. When a caller asks a question and hears nothing back, their brain doesn't fill the silence with patience — it fills it with "this is a machine, and I'm wasting my time."
That's why, when we build Mangal on the open-source Dograh voice stack, we treat latency as a feature. Our target for a natural, conversational response is sub-second — comfortably under the ~800ms point where a caller starts to feel the gap — and we measure every stage of the pipeline against that budget.
Where the milliseconds actually go
An AI voice agent isn't one model. Every reply is a relay race between at least three components:
- Speech-to-text. Turning the caller's words into text — with Indian accents, mixed Hindi–English sentences, and background noise all in play.
- Language model. Deciding what the agent should say next, given the conversation so far and the business rules it's been configured with.
- Text-to-speech. Producing natural-sounding speech that doesn't sound read from a card.
Add network round-trips and orchestration overhead, and it's easy to leak 2–3 seconds before a caller hears a word. The art is in where you spend the budget: streaming output instead of waiting for a full response, warm models instead of cold starts, and keeping the pipeline as short as possible without sacrificing voice quality.
What happens when you blow the budget
Above roughly 800ms–1s, callers start interrupting, repeating themselves, or just hanging up. That's not a voice-quality problem — the words might be perfect. It's a trust problem. Humans have precise instincts about conversational timing, and once those instincts say "this isn't a person," everything else the agent says is filtered through that doubt.
For outbound calls — a lead being called back, a payment reminder, a follow-up — the stakes are even higher. The caller didn't ask for this conversation, so the first few seconds decide whether it continues at all. A fast, fluid opener is what earns the next ten seconds of the caller's attention.
How we keep it sub-second
- Streaming, not batching. The agent starts speaking as soon as the first tokens are ready, instead of waiting for a complete sentence.
- Short, efficient pipelines. Fewer hops between services means fewer places for latency to hide.
- Measured on every call. We watch real response times, not lab benchmarks — a demo in a quiet office is not the same as a receptionist's desk in a clinic.
- Language-aware configuration. Hindi and Hinglish phrasing is tuned so the agent doesn't need extra thinking time for code-switching.
The takeaway
Voice AI products are compared on features, but they're experienced on timing. Get the latency budget right and the caller forgets they're talking to an agent. Miss it, and no amount of accuracy or vocabulary will win them back. If you're evaluating an AI calling platform, ask one question first: how long does the caller wait before hearing a reply? Everything else follows from there.
Want to hear the difference for yourself? Try Mangal on a live call — no card required.