How to Build an AI Voice Agent (2026 Guide)

You can make an AI answer a phone number in an afternoon. That is the demo.

The production job is harder: the caller interrupts, says “next Friday” in a noisy car, the calendar API times out, and then asks for a person. A useful agent must recover without inventing an appointment or trapping the caller in a loop.

This guide shows the path from that first call to a narrow, monitored production deployment.

Quick answer: Start with one inbound job, a new test number, and a managed platform such as Vapi or Retell. Give the agent one or two server-side tools, make human transfer an explicit escape route, and launch to a small share of calls. Do not begin with cold outbound calling or a fully autonomous “front desk.”

The architecture: a real-time loop with escape routes

A voice agent has five working layers:

1. Telephony connects a phone number to the application.
2. Turn detection and speech recognition decide when the caller has finished and convert speech into usable input.
3. The model chooses a short response or a tool call.
4. Text-to-speech produces audio while interruption handling, often called barge-in, stops playback when the caller speaks.
5. Your application reads calendars or customer records, writes approved changes, transfers the call, and stores events for monitoring.

Managed platforms package much of this loop. A custom stack can instead connect a carrier such as Twilio to a real-time model; OpenAI's Realtime API supports WebRTC, WebSocket, and SIP connections, along with voice activity detection and automatic response interruption. The custom route provides more control, but your team owns media streaming, retries, observability, and on-call failures.

1. Choose one job and a platform

Write the first release as a sentence: “Answer inbound calls after hours, answer approved FAQs, and create a callback request.” That is testable. “Handle reception” is not.

Use a managed platform when speed matters and its telephony, model, voice, and transfer options fit. Vapi separates its platform charge from model, voice, transcription, and carrier costs; its pricing page explains the usage-based model. Retell publishes a configurable component table on its voice-agent pricing page. Neither is automatically cheaper for every configuration.

Choose a custom carrier-and-model stack when you need an unsupported region, unusual media routing, strict infrastructure control, or economics proven at meaningful volume. It is usually a poor first prototype because every saved platform feature becomes engineering work.

Also choose the model and voice together. A larger model may handle more complex instructions better in your tests, but it may cost more or respond more slowly. A premium voice may sound better but does not repair a bad workflow. Test names, addresses, dates, and your industry's vocabulary before selecting on a polished sample alone.

2. Provision a number without risking the main line

Buy a new local or toll-free test number in the target country. Confirm inbound and outbound rates, number rental, concurrency limits, and whether emergency calling is supported. Prices differ by destination and number type: for example, Twilio's U.S. list price on July 11, 2026 was $1.15/month for a local number and $0.0085/minute for inbound local calls.

Do not port the main business number during the experiment. After acceptance testing, use carrier forwarding or route only a controlled condition—after hours, overflow, or one campaign—to the agent. Keep a route that can send calls back to voicemail or a human queue if the AI service is unavailable.

For a managed build, the minimum end-to-end path is: provision the test number; attach it to an inbound agent; configure the prompt, voice, turn timing, and interruption behavior; register an authenticated HTTPS tool endpoint; verify the provider's webhook signature and call ID before processing a request; return a bounded result; configure transfer and carrier fallback; then place real test calls. The exact dashboard labels will change, but these artifacts and trust boundaries will not.

3. Design turn-taking before polishing the voice

Three settings shape whether a call feels usable:

• End-of-turn timing: too eager and the agent cuts off thoughtful callers; too slow and every response has an awkward pause.
• Barge-in sensitivity: too sensitive and background noise cancels speech; not sensitive enough and the caller cannot interrupt.
• Response length: spoken answers should usually be one or two sentences, followed by a question.

Test these with actual phone audio, not only a laptop microphone. OpenAI's Realtime documentation, for example, notes that shorter silence detection responds faster but may jump in during brief pauses. There is no universal “sub-second” setting that works for every accent, network, and environment.

4. Write an operational prompt

The prompt should define identity, scope, sources, confirmation rules, escalation, and spoken style. It should not contain credentials or replace authorization checks in code.

You are River Dental's automated scheduling assistant.
Say that you are an AI assistant in your first sentence.

Goal: help with opening hours, available appointments, and callback requests.
Use only the supplied clinic facts and tool results.
Never provide diagnosis, clinical advice, or an unverified price.

Before creating a booking, repeat the date, time, service, caller name,
and callback number. Ask for a clear yes. If a tool fails, do not claim
success; offer a human transfer or callback.

If the caller requests a person or sounds distressed, offer the configured
human route. If the caller describes an urgent dental or medical situation,
say you cannot provide emergency help. Follow only the clinic-approved local
emergency instructions. Transfer only to a staffed, configured emergency
route. If it is unavailable, give the approved local emergency route; never
substitute a routine callback or voicemail.

Speak in short sentences. Ask one question at a time.

Keep volatile facts—availability, account status, prices—outside the prompt. Retrieve them through a tool or a controlled knowledge source. Otherwise yesterday's prompt becomes today's confident mistake.

5. Give tools narrow contracts

A tool call is an untrusted request from the model. Validate it on your server, authorize it against the caller and account, make writes idempotent, and return a small machine-readable result.

{
  "type": "function",
  "name": "create_callback",
  "description": "Create a callback only after the caller confirms the details",
  "parameters": {
    "type": "object",
    "additionalProperties": false,
    "properties": {
      "callerName": { "type": "string", "minLength": 1, "maxLength": 100 },
      "phoneE164": { "type": "string", "pattern": "^[+][1-9][0-9]{7,14}$" },
      "reason": {
        "type": "string",
        "enum": ["appointment", "billing", "records", "other"]
      },
      "confirmed": { "type": "boolean", "const": true }
    },
    "required": ["callerName", "phoneE164", "reason", "confirmed"]
  }
}

The endpoint returns one of these bounded response objects after the server attempts the write:

{ "status": "created", "referenceId": "cb_123", "expectedWindow": "within one business day" }

{ "status": "unavailable", "safeMessage": "I couldn't save that request. I can transfer you now." }

The schema requires confirmed: true, but that is not authorization. Reject the write unless server-side identity, account authorization, and business rules also pass. Reject extra properties and invalid phone numbers before application code runs. Use an idempotency key based on the verified provider call ID and request so a retry does not create duplicates. Set a short timeout; after it expires, the agent should follow the failure message, not improvise. Log tool name, duration, result, and reference ID—but redact secrets and sensitive fields.

Build transfer as a first-class tool too. Define the destination, business-hours behavior, what happens if nobody answers, and the short context the human receives. Never promise “someone is joining now” until the transfer has actually connected.

For this dental example, have the clinic's qualified clinical and safety reviewers approve jurisdiction-appropriate emergency wording and routes before launch. A normal front-desk transfer, callback, or voicemail is not an emergency fallback, and this implementation example is not legal or clinical advice.

6. Handle consent and disclosure as product requirements

This is an engineering checkpoint, not a line of magic wording. Rules depend on call direction, purpose, jurisdiction, recording, industry, and how consent was obtained.

For U.S. outbound calls, the FCC has ruled that AI-generated voices fall under the TCPA's restrictions on artificial or prerecorded voices. Its declaratory ruling says prior express consent is required unless an exemption applies, with identification/disclosure requirements and specified opt-out methods for advertising or telemarketing calls. Marketing may trigger stricter consent requirements.

For inbound calls, a brief opening such as “I'm River Dental's AI assistant” is clear product behavior, but it is not a substitute for reviewing applicable law. Recording and transcription consent varies by location. Before launch, have qualified counsel review the exact countries or states, script, recording settings, retention, opt-out flow, and evidence of consent. Provide a human route and honor do-not-call or deletion requests through a documented process.

7. Test failure, not just conversation

Use a repeatable matrix and save the expected outcome before each call.

Review transcripts, recordings where permitted, tool logs, and the resulting business record. A fluent transcript with the wrong calendar write is still a failed test.

What does a lean voice agent cost?

Treat these as planning examples in U.S. dollars, not quotes. Pricing was checked July 11, 2026 and can change.

For a concrete monthly example, assume 100 calls × 4 minutes = 400 minutes. At $0.088/minute, usage is $35.20. Add one $2 number for $37.20/month, before SMS, recordings, add-ons, support, taxes, transferred call legs, or your own application hosting. The rate comes from Retell's published component prices on the date above; another model or voice changes the arithmetic.

Do not confuse the prototype budget with production readiness. Production cost also includes integration maintenance, incident response, transcript review, compliance work, and staff time handling escalations.

Launch gradually and monitor outcomes

Start with internal callers, then a small slice of low-risk inbound traffic. Expand only when the failure rate and human workload are acceptable.

Track more than minutes:

• call connection and completion rate;
• transfer rate, transfer success, and callback backlog;
• tool success, timeout, and duplicate-write rate;
• p50 and p95 response latency;
• interruption recovery and repeated-prompt rate;
• confirmed task completion, checked against the source system;
• opt-outs, complaints, safety escalations, and cost per completed task.

Sample failed and successful calls every week. Set alerts for provider errors, rising latency, tool failures, spend, and zero-call anomalies that may mean routing is broken.

Before opening traffic, verify: number routing; AI disclosure; consent evidence where required; recording settings; prompt version; tool authorization; transfer and voicemail fallbacks; data retention; redaction; dashboards; alerts; spend limits; and a named person who can disable the agent.

The safest first voice agent is deliberately boring. It handles one repeatable job, confirms every write, and gets out of the way when it is uncertain. If you want help defining that narrow first release, talk to SprintX about an AI voice agent build.