Season 4 Ep 9 — In Ep 8, audio was one modality among many. Ep 9 focuses on a single product category: voice. Real-time behavior, naturalness, and safety — why holding all three at once is hard, and how to do it.
- Prologue — The Year of "AI Without a Screen"
- Chapter 1 · Two Architectures
- Chapter 2 · Designing the Real-Time Pipeline
- Chapter 3 · Designing the Latency Budget
- Chapter 4 · Speech LLMs — GPT-4o Realtime, Gemini Live, Moshi
- Chapter 5 · Emotion, Prosody, Speed — Expressive Speech
- Chapter 6 · Telephony (PSTN), Browser, Mobile
- Chapter 7 · Deepfake and Voice-Cloning Threats, and Defenses
- Chapter 8 · What Makes Korean-Language Voice Products Different
- Chapter 9 · Cost and Operations
- Chapter 10 · Three Real-World Cases
- Chapter 11 · Twelve UX Principles
- Chapter 12 · Ten Antipatterns
- 12.1 Shipping Without Measuring Latency
- 12.2 Overly Long Prompts for the LLM
- 12.3 No Interruption Handling
- 12.4 No Consistency in Emotion and Tone
- 12.5 Trusting Deepfake Detection Alone
- 12.6 Missing Legal Notices
- 12.7 Implementing Real Time with Batch TTS
- 12.8 Sensitive Transactions by Voice Alone
- 12.9 Ignoring Dialects and Elderly Voices
- 12.10 No Logs
- Chapter 13 · Checklist — 12 Items Before Launching Voice AI
- Chapter 14 · Next Up — Season 4 Ep 10: "LLM Security"
Prologue — The Year of "AI Without a Screen"
OpenAI's GPT-4o demo in May 2024 was the turning point for voice AI. Latency of ~300ms, natural pronunciation, emotional expression, interruption — a single end-to-end model delivered quality that a traditional STT→LLM→TTS pipeline struggles to reach.
As of 2025:
- Contact centers: companies where voice AI handles 50%+ of simple inquiries
- Education and coaching: the standard UX for English and foreign-language conversation apps
- Health and counseling: therapeutic conversation support (medical advice is still off-limits)
- Personal assistants: cars, smart homes, wearables
This post pulls together, in one pass, everything you need to know when you build voice AI.
Chapter 1 · Two Architectures
1.1 The Traditional Pipeline (STT → LLM → TTS)
[Mic] → VAD → [Streaming STT] → [LLM] → [Streaming TTS] → [Speaker]
- Each stage is independent → flexible, easy to debug
- Free choice of models and vendors
- Downside: latency accumulates (500ms–1.5s), emotion and prosody are lost
1.2 Speech LLMs (end-to-end)
[Mic] → [Speech LLM: speech → speech] → [Speaker]
- Audio tokens go straight in and out
- Reproduces emotion, laughter, even sighs
- Latency ~300ms
- Downside: large models, a limited set of vendors (OpenAI/Google/Moshi and a few others), hard to control in fine detail
1.3 The Hybrid Reality of 2025
- Real-time two-way conversation → speech LLM
- Non-real-time and batch processing → traditional pipeline (cost and logging advantages)
- Enterprise: compliance and audit demands are strong, so the traditional pipeline still dominates
- Consumer: speech LLMs are spreading fast
Chapter 2 · Designing the Real-Time Pipeline
2.1 VAD (Voice Activity Detection)
- Purpose: send only the segments where someone is speaking to STT, cutting cost and latency
- Models: Silero VAD (open), WebRTC VAD, PyAnnote
- Parameters: threshold, minimum utterance length, silence-based end detection
2.2 Streaming STT
- Emits partial transcripts every 250–500ms
- The final transcript arrives when end-of-utterance is detected
- Deepgram, AssemblyAI, Whisper (streaming), Google/Azure/AWS, Clova/Kakao
2.3 LLM Processing
- You can "start thinking" from the partial transcript (build context in advance)
- Generate the response once the final transcript is confirmed
- Speculative generation: start the answer from the partial, then correct it once the final arrives
2.4 Streaming TTS
- Play back sequentially at sentence boundaries
- Natural gaps without "thinking" fillers (uh/hmm)
- The core of latency reduction
2.5 Interruption (Barge-in)
- The moment the user starts speaking, stop TTS immediately
- A response that was only partially played should be folded into the next turn as a "summary/correction"
- Making the cut-off feel human rather than robotic is what UX lives or dies on
Chapter 3 · Designing the Latency Budget
3.1 The Target
A figure frequently cited in research and in the field: the average gap between turns in human conversation is ~200ms. If the AI answers within 500ms–1s it feels natural. Past 1.5s it feels awkward.
3.2 An Example Budget (Traditional Pipeline)
VAD end-of-utterance detection: 150ms
STT final result: 150ms
LLM TTFT (first token): 300ms
TTS first audio chunk: 150ms
Network and decoding: 100ms
───────────────────
Total to first audio playback: about 850ms
3.3 Where to Cut
- Use a small/fast LLM as the first responder (fillers, acknowledgements)
- Streaming: for STT, LLM, and TTS alike
- Speech LLMs: the architecture itself is shorter
- Locality: deploy in a region close to the user
3.4 Jitter and Stability
- p95/p99 matter more than the average
- One long turn wrecks the rhythm of every turn that follows → users drop off
Chapter 4 · Speech LLMs — GPT-4o Realtime, Gemini Live, Moshi
4.1 GPT-4o Realtime
- Real-time and bidirectional across speech and text
- Built on WebSocket/WebRTC
- Latency 300–500ms
- Considerably more expensive than text, but latency and quality are overwhelming
4.2 Gemini Live
- 1M context plus real-time multimodal
- Can combine screen sharing and camera input
- Excellent at interruption and emotional expression
4.3 Moshi (Kyutai, open)
- An open-source speech LLM
- Full-duplex, low latency
- Can run locally — attractive for privacy-oriented products
4.4 Differences
| Item | GPT-4o | Gemini Live | Moshi |
|---|---|---|---|
| Price | High | Medium | Self-hosted |
| Korean | Excellent | Excellent | Limited |
| Customization | Limited | Limited | High (open) |
| Modality mix | Speech, text | Speech, video, text | Speech-first |
| Enterprise | API + logging | Google Cloud | Self-managed |
Chapter 5 · Emotion, Prosody, Speed — Expressive Speech
5.1 The Control Axes of TTS
- Tone/voice: persona, gender, age
- Emotion: neutral, happy, sad, angry, excited, and so on
- Speed: 0.8x–1.5x
- Stress: emphasis on particular words
- Breath and pauses: SSML such as
<break time="300ms"/>
5.2 SSML (Speech Synthesis Markup Language)
<speak>
Hello, <break time="300ms"/>
today we are going to talk about something <emphasis level="strong">very important</emphasis>.
</speak>
- Supported by most vendors, including Google, Microsoft, Naver Clova, and Kakao
- ElevenLabs and OpenAI TTS use their own formats (curly-brace tags, natural-language directions)
5.3 The Politics of Emotion
- Corporate brand tone: keep excessive emotion out
- Services for children: warm tone, slower speed
- Medical and counseling: neutral and calm
Do not use the same tone for every voice. Split the profiles to match the service context.
Chapter 6 · Telephony (PSTN), Browser, Mobile
6.1 Telephony Integration
- Twilio, Vonage, Plivo, SignalWire: programmable telephony
- A SIP/PSTN gateway hands the voice stream to STT/LLM
- Korea: local solutions such as LG U+ AI Call, KT AICC, and Naver Cloud AiCall
6.2 The Browser
- Two-way audio over WebRTC
- Permissions, echo cancellation, and noise suppression are built in
- Checking mobile browser compatibility is mandatory
6.3 Mobile Apps
- iOS: AVAudioEngine, Speech, AVSpeechSynthesizer
- Android: AudioRecord, MediaCodec, TTS API
- Native VAD is available
6.4 Automotive and Embedded
- Compensating for cabin and wind noise
- Safety while driving: short emergency responses
- Offline mode is mandatory (connectivity is unstable)
Chapter 7 · Deepfake and Voice-Cloning Threats, and Defenses
7.1 The Threats
- Voice phishing: cloning a family member's voice to demand an urgent transfer
- Executive impersonation: transfer instructions in the CEO's voice
- Identity spoofing: bypassing identity verification at call centers and financial institutions
7.2 Defense Layers
- Liveness detection: prepared lines vs. demanding a live response
- Voice biometrics: voiceprint plus behavior, in combination
- Deepfake detection: synthetic-speech classifiers (accuracy in the 90% range, not perfect)
- Callback verification: for sensitive requests, call back on the registered number
- Mandatory AI disclosure: laws in some countries and states (notice when the voice is AI)
7.3 Watermarking
- Embed an imperceptible watermark in AI-generated speech (for example, Google SynthID for audio)
- A detector identifies AI provenance by checking the watermark
- Not yet 100% reliable — treat it as one defense layer among several
7.4 Operational Policy
- Sensitive transactions are "no AI voice" → hand off to a human
- Retain and audit voice logs (within what the law allows)
- AI notice when a new call connects: "This call is being handled by an AI agent."
Chapter 8 · What Makes Korean-Language Voice Products Different
8.1 STT
- Accuracy on standard Korean is 95%+ with Clova/Kakao
- Coverage of regional dialects is still lacking
- Custom dictionaries are mandatory for specialist terminology (medicine, law)
8.2 TTS
- Naver CLOVA Voice, Kakao i, Supertone: top tier for naturalness
- ElevenLabs Korean voices are improving rapidly as well
- Controlling Korean-specific intonation and honorific speech matters
8.3 Law and Regulation
- Telecommunications Business Act and E-Commerce Act: call-recording notice is mandatory
- Personal Information Protection Act: voice data can also contain sensitive information
- Finance: AI voice alone cannot authenticate a customer for KYC
8.4 Culture and UX
- Consistency between casual and honorific speech
- Brand policy on forms of address (-ssi / -nim / gogaek-nim)
- Opening a response with something like "Yes, Mr./Ms. So-and-so" raises perceived trust
Chapter 9 · Cost and Operations
9.1 Cost Structure
- STT: $0.003–0.02 per minute
- TTS: $0.00001–0.00005 per character (depending on voice and quality)
- LLM: per token (input/output) — same as text
- Speech LLMs: tend to be 2–5x more expensive than a traditional pipeline (early 2025)
- Lines (PSTN): an additional per-minute cost
9.2 Scaling
- N concurrent calls → a TCP/WebSocket per stream plus an LLM slot
- Orchestration (Kubernetes + HPA) plus regional distribution
- A warm pool to reduce cold starts
9.3 Observability
- Per-call latency (p50/p95), interruption frequency, rate of transfer to a human
- Real-time monitors for hallucination and inappropriate statements
- Tracking NPS and resolution rate
Chapter 10 · Three Real-World Cases
10.1 First-Line Contact Center Response
- Use cases: reservations, address changes, simple FAQs
- Complex inquiries get a smooth handoff to a human agent (with a context summary passed along)
- Automatic checks on recording and compliance phrasing
10.2 An English Conversation Learning App
- Natural conversation via a speech LLM (GPT-4o realtime)
- Pronunciation scoring (traditional STT is more accurate here)
- Difficulty tuned to the learner's level
10.3 Senior Care and Wellbeing
- Companionship, simple reminders, medication prompts
- Warm emotional tone, slower speed
- Contact a guardian when an emergency (a fall, unusual symptoms) is detected
Chapter 11 · Twelve UX Principles
- Latency is everything: past one second it feels awkward
- Interruption: stop the instant the user cuts in
- Keep it short: three to five sentences per turn
- Signal uncertainty: when you don't know, say "I'm not sure about that"
- Anti-spoofing: never allow sensitive actions by voice alone
- Restrained emotion: brand tone first, no overacting
- Fillers: short signals like "mm" and "right" convey "I'm listening"
- Repeat requests: when the user says "what?", adjust speed and pronunciation
- Closing signal: wrap up naturally with "Is there anything else I can help with?"
- Error recovery: if STT looks wrong, ask back with "Did I understand that correctly?"
- Block personal data: discourage speaking card numbers or national ID numbers
- Accessibility: design separate UX for deaf and elderly users
Chapter 12 · Ten Antipatterns
12.1 Shipping Without Measuring Latency
90% of perceived quality is latency. p95 is the yardstick.
12.2 Overly Long Prompts for the LLM
Real-time behavior breaks. Keep the system prompt short and sharp.
12.3 No Interruption Handling
A robot that keeps pushing while you're talking. It destroys the UX.
12.4 No Consistency in Emotion and Tone
Mixing three voices and confusing the brand.
12.5 Trusting Deepfake Detection Alone
Detectors are imperfect. Combine liveness, callback, and policy.
12.6 Missing Legal Notices
Call-recording and AI-response notices are mandatory.
12.7 Implementing Real Time with Batch TTS
Needs a rewrite to chunk-level streaming.
12.8 Sensitive Transactions by Voice Alone
Identity verification in finance and healthcare needs an additional channel.
12.9 Ignoring Dialects and Elderly Voices
STT accuracy varies widely. Diversify samples and customize.
12.10 No Logs
You lose the evidence for disputes and for quality improvement.
Chapter 13 · Checklist — 12 Items Before Launching Voice AI
- p50/p95 latency targets and measurement
- Failure fallbacks for each stage: VAD/STT/LLM/TTS
- UX testing for interruption handling
- Defined emotion and tone profiles
- AI-response notice inserted
- Recording, retention, and deletion policies compliant with the law
- Deepfake and spoofing defense layers
- Human handoff triggers and context transfer
- Measuring user satisfaction (NPS/resolution rate)
- Recommending non-voice channels for sensitive transactions and data
- Verified Korean STT/TTS quality (vendor comparison)
- Alternative paths for accessibility (hearing-impaired, elderly)
Chapter 14 · Next Up — Season 4 Ep 10: "LLM Security"
For voice and for text alike, the biggest threat to a 2025 LLM product is a security incident.
- Twelve variants of prompt injection
- Jailbreaks and roleplay bypasses
- Data exfiltration vectors
- Model extraction and parameter leakage
- Denial of service (DoS) and billing attacks
- Building and automating red teams (PyRIT, Garak)
- Guardrail prompts vs. classifier models
- Privacy and PII leakage
- Supply-chain threats (models, MCP servers)
- Standards and regulation (EU AI Act, Korean privacy law)
- The incident response playbook
"Security is not a feature. It is a default." And yet through 2024–2025 many LLM products still fail to meet even the defaults.
See you in the next post.
Summary: Voice AI is a three-beat rhythm of real-time behavior, naturalness, and safety. The choice between a traditional pipeline and a speech LLM comes down to your latency, control, and compliance demands, and p95 latency decides 90% of the user experience. Manage emotion, speed, and tone through brand policy, and treat deepfakes and spoofing as something only layered defense handles. Korean-language products push the quality frontier by combining local assets like Clova, Kakao, and Supertone with global speech LLMs. In the era of "AI without a screen," the limit is now the imagination of the person building it.