A voice assistant that takes the call, even when no one's at the desk

The Problem

Phone calls are the highest-context but most-expensive support channel. A hostess takes a reservation booking; a clinic receptionist handles appointment scheduling; a call-center agent processes survey responses. The work is repetitive: a fixed set of questions, slots and outcomes. But each call eats five or ten minutes of human time, and most operations only run during business hours.

Conventional IVR (the “press 1 for sales” tree) is universally hated. The phrase “speak to an agent” is the most-spoken phrase to most IVR systems. The reason: rigid menus that don’t match how customers actually phrase their needs.

LLM-powered voice assistants change the shape. Customers speak naturally; the assistant understands intent, fills in the relevant slots, and either completes the action or routes to a human with the context preserved. The technology matured enough in the last two years that production voice deployments are now realistic.

What the Solution Does

A voice-first AI assistant that takes phone calls and handles structured-outcome workflows.

1. Inbound or outbound, accept calls from customers, or place calls to them on a schedule.
2. Natural-language understanding, the assistant reads intent (create booking, edit booking, cancel booking) from how the customer speaks. There are no menu selections.
3. Slot-filling conversation, the assistant asks for the information it needs: date, time, party size, preferences. It answers follow-up questions and goes back to fill missing slots.
4. Action, once slots are filled and validated, the assistant acts through your operational system. It creates the booking, updates the system, or schedules the callback.
5. Handoff when appropriate, complex cases, exceptions, or explicit customer requests route to a human operator with full context.

Where It Fits

This makes sense if you…

• Operate phone-call-heavy workflows with structured outcomes (booking, appointment scheduling, basic support, outbound surveys)
• Have measurable cost from call volume, staff time, missed-call lost-revenue, after-hours gaps
• Have backend systems that can be integrated for taking action (booking systems, calendar APIs, etc.)
• Are willing to invest in voice-specific tuning (customer phrasing varies more in voice than text)
• Will design the handoff carefully, bad voice handoff is worse than bad text handoff

This is probably not the right time if you…

• Have unstructured conversation flows where slot-filling doesn’t map (“tell me about my experience”, not a slot-filling job)
• Need every call answered by a human for relationship reasons (luxury / high-touch contexts)
• Cannot integrate with backend systems, voice is the front-end; backend integration is mandatory
• Operate in highly-regulated voice contexts (healthcare appointment scheduling has rules; broader healthcare interaction may be restricted)

Business Value

24/7 coverage. The biggest single value: calls get answered at 11pm, on Sunday, during the holiday, when no human is at the desk. Missed-call lost revenue drops to zero on the scenarios the voice bot covers.

Agent time recovery. When the voice bot handles 60-70% of routine calls, agents handle only the complex cases. Throughput per agent goes up; CSAT on the complex cases (which now get more attention) often improves.

Consistency. Every caller gets one fixed set of questions in one fixed order. Information capture is consistent. No “the agent forgot to ask about dietary restrictions” cases.

Outbound at scale. Outbound voice automation, appointment reminders, survey follow-ups, scheduled callbacks, is hard to staff economically. Voice bots run them at scale with predictable cost per call.

How It Works

The architecture below is what powers our working restaurant-booking demo, and what we adapt for other voice deployments.

1. Voice interface (Voice UI)

Inbound: telephony provider (Twilio, RingCentral, or similar) routes calls to our system. Outbound: the system places calls through that telephony provider on a schedule or trigger.

ASR (automatic speech recognition) converts the customer’s speech to text. We use cloud ASR providers (Google, Deepgram, ElevenLabs depending on language and quality requirements) or self-hosted alternatives.

TTS (text-to-speech) converts the assistant’s response back to speech. Voice quality varies sharply by provider and language, we benchmark for each deployment.

2. NLU, understanding intent

For each customer utterance, the NLU component identifies intent (create / edit / cancel booking) and any slot values present. We use LLM-backed NLU (better than rigid intent classifiers for natural speech) plus structured slot extraction.

3. LLM, conversation management

The conversation manager (LLM-based) decides what to ask next. Given the current slot-fill state and the customer’s last utterance, it generates the next question, either to fill missing slots or to confirm before action.

We pre-brief the LLM with the slot schema for the scenario: “you’re booking a restaurant table; ask for date, time, party size, and any seating preferences; confirm before booking”. The LLM handles natural variations gracefully without scripted decision trees.

4. Backend integration

When the assistant is ready to take action, it calls the integrated backend, booking system, calendar API, CRM. Validation happens here too (is the date in the future? is the requested time available? is the party size within capacity?).

5. Handoff and exception handling

When the assistant is stuck, slot can’t be filled after retries, customer asks for a human, scenario doesn’t match, the call routes to a human operator with the conversation context.

6. Continuous improvement

Call transcripts and outcomes feed back. Patterns that the assistant handles poorly get addressed via prompt refinements or slot-schema updates. Datapipe handles the data pipeline.

Stack

Telephony: Twilio / RingCentral / equivalent. ASR: Google Speech / Deepgram / Whisper (self-hosted). TTS: ElevenLabs / Google / Azure. LLM: OpenAI / Anthropic / open-source (Llama, Qwen). NLU: LLM-backed with explicit slot schema. Backend integration: REST APIs to your operational systems. Operator workplace: Chatwoot or your existing system.

What You Need to Make This Work

Data. Scenario definitions, intents (booking, support, etc.) and slots per scenario. Examples of typical customer phrasing help us tune the NLU.

Integrations. Telephony provider (or we set up). Backend systems for action (booking, CRM, calendar). Operator workspace for handoff. Audit log destination.

Hardware. Cloud-only. Telephony is provider-based.

Team. A workflow lead who knows the conversation scenarios. A backend integration contact (~20-30 hours during pilot). A pilot user group for the deployment.

Implementation Roadmap

1. Test (4-6 weeks)

Pick one scenario (restaurant booking, appointment scheduling, etc.). Define slot schema. Set up telephony and ASR/TTS. Build the LLM-backed conversation manager. Test on a representative call set. Output: working voice assistant for one scenario with measured slot-fill success rate.

2. Pilot (2-4 months)

Production deployment for that scenario on a slice of inbound (or outbound) volume. Tune prompts. Wire up handoff. Build dashboards. Measure conversion and handoff rates. Output: working production deployment with documented business outcomes.

3. Production (4-6 months)

Expand to additional scenarios / additional volume. Continuous improvement via call transcript review and prompt updates. Your team owns the operational workflow; we stay on for prompt tuning and integration extensions.

Keep in Mind

Where it breaks, and what we tell you up front:

• Voice is harder than text. ASR errors, accent variation, background noise, telephony quality, all degrade. Test on your real call quality. Lab recordings flatter the system.
• Latency budget is tight. Customer speaks; the ASR, then LLM, then TTS round-trip needs to be under 2 seconds for natural conversation. Long latencies make the bot feel broken.
• Out-of-script utterances need real care. “Actually can you also tell me your hours?”, that’s outside the booking scenario. The assistant should either answer (with a knowledge base) or politely redirect. It should never get confused.
• Language coverage varies sharply. ASR/TTS quality for English is excellent; for Russian and major European languages it’s good; for less-common languages it’s lower. We benchmark per deployment.
• TTS voice quality is a brand decision. A robotic voice signals “this is automation”. A near-natural voice signals “this is professional”. Test both with your customer base.
• Compliance varies by jurisdiction. Recording disclosure, voice-recognition consent, local laws apply. We work through these during pilot.
• Bad handoff is worse than bad text handoff. A frustrated caller getting transferred to “let me start over with a human” is one of the worst CX moments. We over-invest in context preservation through the handoff.

FAQ

Can we use this for outbound calls (reminders, surveys)?

Yes, that’s a major use case. Outbound is often easier than inbound because the calling context is known upfront (you know who you’re calling and why).

What about IVR replacement on existing phone numbers?

Yes. The voice bot can replace the IVR layer entirely or sit alongside it. Most production deployments start as a partial replacement (one scenario at a time) and expand.

How does the bot handle people who insist on speaking to a human?

The handoff path is designed for it. The bot recognizes the request (explicit “human” / “agent” or implicit signals like frustration), confirms briefly, and routes to a human operator with context. We tune this carefully, it’s the single biggest CX risk in voice deployments.

Can the bot make payments?

Read-only and low-risk actions: yes. Payments and high-risk actions: typically routed to a human or to a verified-channel system. PCI compliance and customer comfort both push this way.

What languages do you support?

We’ve shipped English and Russian. Spanish, German, French are well-supported by current ASR/TTS providers. Less-common languages have lower-quality voice stacks but are doable. We benchmark per language.

How is this different from off-the-shelf voice AI (Voiceflow, Bland, Vapi)?

Commercial voice AI platforms are excellent for organizations that want a ready-to-run product with managed telephony. Our approach is the right choice when you need: custom backend integration, fully on-prem ASR/TTS, very specific compliance requirements, or substantially lower per-call cost at high volume.

Ready to Discuss?

If you operate phone-call-heavy workflows with structured outcomes and after-hours coverage or agent throughput is a real cost, this is a worthwhile pilot. We’ll walk through your call scenarios, your backend systems, and your handoff needs, and tell you what to expect from a single-scenario pilot.

Discuss your project