LabHub

Blog

Voice AI in 2026 — ElevenLabs / Cartesia / Sesame / Whisper Turbo / Deepgram / Parakeet Deep Dive

2026-05-15 · 22 min read · #voice-ai#tts#stt#asr#elevenlabs#cartesia#sesame#whisper#deepgram#parakeet#assemblyai#voicevox#vapi#retell#deep-dive#english

한국어English日本語

Voice AI 2026 series — Since Whisper Turbo arrived in October 2024, voice AI has been moving as fast as text LLMs. This post is the May 2026 map of TTS, STT, and realtime voice agents.

Prologue — Why Voice, Now Again

The 2022–2023 LLM boom was text-first. ChatGPT web chat, GitHub Copilot, RAG bots — all keyboard input. Voice was "yeah, someday maybe."

By 2026 three events changed the picture.

  1. Whisper Large v3 Turbo (Oct 2024) — OpenAI open-sourced a turbo variant that's 8x faster than v3 large. Realtime STT on a single A100 became practical.
  2. Cartesia Sonic 2 (2024) — A startup founded by the Mamba state-space model authors (Albert Gu, Tri Dao). Sub-90ms TTS that's fast enough that pairing it with a GPT-4-class LLM makes the user not realize they're talking to a bot.
  3. Sesame (March 2025) — Oculus co-founder Brendan Iribe's "voice presence" demo. A 30-second clip on social media convinced everyone "this is different."

Layer in ElevenLabs V3, Deepgram Nova-3, AssemblyAI Universal-2, NVIDIA Parakeet 1.1, OpenAI Realtime API, plus voice-agent platforms like Vapi and Retell — and by May 2026, "AI call center" is no longer a PoC. It ships.

This post maps that landscape across 14 chapters.

Chapter 1 · The 2026 Voice AI Map — TTS / STT / Voice Agents

1.1 The Three Axes

A voice AI system almost always splits into three components.

StageRoleRepresentative models/services
STT (Speech-to-Text)human speech to textWhisper Turbo, Deepgram Nova-3, AssemblyAI Universal-2, Parakeet
LLMtext input to text responseGPT-4o, Claude 3.5, Gemini 2
TTS (Text-to-Speech)text to speechElevenLabs, Cartesia Sonic 2, Sesame, OpenAI TTS, VOICEVOX

Then there are unified systems that handle all of it in one model: OpenAI Realtime API, Google Live API, ElevenLabs Conversational v2. Unified is more natural but pricing, constraints, and debuggability differ.

1.2 Evaluation Axes

In 2026, voice AI is evaluated on four axes.

No single model wins all four. So the answer depends on the workload. Call center vs game NPC vs audiobook all have different priorities.

1.3 Open Source vs Commercial

AxisOpen sourceCommercial
TTS qualityF5-TTS, XTTS-v2, ChatTTS — got better, not at the topElevenLabs, Cartesia, Sesame — dominant
STT accuracyWhisper, Parakeet — basically caught upDeepgram, AssemblyAI — slight edge, domain tuning
Latencyself-host can reach 100ms200–500ms (network)
CostGPU only0.01–0.01–0.30 per minute

Open source has basically caught commercial STT. The TTS gap is still real. That's the big picture in 2026.

Chapter 2 · Whisper Large v3 Turbo (Oct 2024) — 8x Faster Multilingual STT

2.1 v3 to v3 turbo

When OpenAI open-sourced Whisper in September 2022, that was one of the biggest single events in voice AI. 99 languages, multilingual, free, commercial-grade STT accuracy.

v3 turbo, released in October 2024, cuts v3 large's decoder layers from 32 to 4 and adds compression. Result:

import whisper

model = whisper.load_model("turbo")  # large-v3-turbo
result = model.transcribe("interview.mp3", language="ko")
print(result["text"])

2.2 Why 8x Matters

v3 large took about 3 minutes on an A100 to transcribe one hour of audio. Realtime was out of reach (streaming was a separate concern).

Turbo finishes the same audio in 22 seconds. Consequences:

2.3 Limitations

2.4 Variants — faster-whisper / WhisperX / Distil-Whisper

ToolCoreUse case
OpenAI Whisper (official)Reference PyTorchResearch/eval
faster-whisperCTranslate2 backend, 4x more speedupProduction batch
WhisperX+ diarization + word timestampsMedia captioning
Distil-WhisperSmaller distilled variantMobile/edge

In production most teams run faster-whisper or WhisperX. The official OpenAI implementation is for research/eval.

Chapter 3 · Deepgram Nova-3 / AssemblyAI Universal-2 — Commercial STT Battle

3.1 Deepgram Nova-3 — The Latency King

Deepgram's edge is latency. Nova-3 has:

from deepgram import DeepgramClient, PrerecordedOptions

deepgram = DeepgramClient(api_key="...")
options = PrerecordedOptions(model="nova-3", smart_format=True, diarize=True)
response = deepgram.listen.prerecorded.v("1").transcribe_file(
    {"buffer": audio_buffer}, options
)

For call-center bots and live captions where 100ms decides the UX, Deepgram is basically the default.

3.2 AssemblyAI Universal-2 — The Full-Stack Player

AssemblyAI competes on "transcript plus post-processing." Universal-2:

Especially strong when you need not just transcript but chapters, summaries, sentiment — media and podcast use cases.

3.3 Speechmatics — Accent Champion

UK-based, strong on a wide range of English accents (Indian, Australian, Caribbean, Scottish). Wins on global call centers where accent diversity is high.

3.4 NVIDIA Riva — Self-Host Champion

NVIDIA Riva is a self-hosted speech SDK. Used by government, finance, and healthcare where data cannot leave the cluster. Common pattern: serve Parakeet on Riva.

3.5 AWS Transcribe / Azure Speech / Google STT

The three hyperscalers all have STT. Accuracy is slightly behind Deepgram/AssemblyAI, but the advantage is integration with the rest of the same cloud.

3.6 Comparison

ServiceEnglish WERKorean WERLatencyPer-min (USD)Strength
Whisper v3 turbo (self)~5%~8%~1–3sGPU onlyFree, multilingual
Deepgram Nova-3~4%~9%<100ms0.004–0.015Low latency
AssemblyAI Universal-2~4%~10%~300ms0.0065+Post-processing
Parakeet 1.1 (self)~5%N/A~200msGPU onlyOpen source SOTA
Speechmatics~5%~9%~200ms0.007+Accents
AWS Transcribe~7%~12%~500ms0.024AWS integration

Numbers are approximate from public benchmarks. Real numbers vary heavily by domain and audio quality.

Chapter 4 · NVIDIA Parakeet 1.1 — The Open-Source SOTA

4.1 What Parakeet Is

A family of open-source STT models trained by NVIDIA with the NeMo framework. When Parakeet 1.1 dropped in late 2024, the verdict was "open source caught commercial."

4.2 Why It's Fast

Whisper uses a Transformer encoder + decoder. It autoregressively generates tokens over a 30-second audio chunk. Parakeet uses a FastConformer encoder + CTC (or RNN-T) decoder. CTC is not autoregressive — it's a sequence alignment — and it's much faster.

The tradeoff: multilingual coverage is weaker than Whisper. Parakeet 1.1 English is English-specialized. A separate multilingual variant (Canary) exists.

4.3 Self-Hosting with NeMo

import nemo.collections.asr as nemo_asr

asr_model = nemo_asr.models.ASRModel.from_pretrained(
    "nvidia/parakeet-tdt-1.1b"
)
transcripts = asr_model.transcribe(["audio.wav"])
print(transcripts[0])

A single GPU can handle hundreds of audio-hours per minute. License is CC-BY-4.0, commercial-friendly.

4.4 Multilingual Variant — Canary

NVIDIA released a separate multilingual ASR called Canary. Supports English/Spanish/German/French and a few more. Korean and Japanese coverage is limited — Whisper still wins those.

Chapter 5 · ElevenLabs — The TTS Standard

5.1 Why ElevenLabs Won

Since 2023 ElevenLabs has been the de facto TTS standard. Why:

  1. Naturalness — the first model where you stop thinking "AI voice" and start hearing "that person"
  2. Multilingual — same voice in 30 languages with accent preserved
  3. Cloning — voice clone from 1 minute of audio, "Professional Voice Clone" with 30+ minutes
  4. Both API and UX are good — developers integrate in 5 minutes, non-developers use the web app directly

5.2 Model Lineup

5.3 V3 Emotion Tags

V3 lets you sprinkle inline tags into the text to mark emotion.

[excited] Welcome back!
[whispers] I have a secret.
[laughs] That's hilarious.
[sighs] Okay, let's start over.

This is a bigger change than it looks. Previously you had to write SSML to tune prosody. V3 just takes natural-language tags.

5.4 Pricing

More expensive than most alternatives but the quality difference shows up in your workflow, so for games, video, and audiobooks it's the default.

5.5 Limitations

Chapter 6 · Cartesia (the Mamba Authors) — Sonic 2 + Ultra-Low Latency

6.1 Who Built It

Cartesia was founded in 2023 by Albert Gu, Tri Dao, and others — the Mamba state-space model paper authors. Mamba was promoted as a Transformer alternative whose memory/compute scale linearly with sequence length. That fits audio well.

6.2 Sonic / Sonic 2 — 90ms TTS

Cartesia's first model Sonic shipped sub-90ms TTS and got noticed. Sonic 2 (late 2024) added:

from cartesia import Cartesia

client = Cartesia(api_key="...")
audio = client.tts.sse(
    model_id="sonic-2",
    transcript="Hello, glad to meet you.",
    voice={"mode": "id", "id": "your_voice_id"},
    output_format={"container": "raw", "encoding": "pcm_f32le", "sample_rate": 44100},
)

6.3 Why It's Fast

Mamba-style state-space models handle token-to-token dependency in O(n) time, unlike Transformer attention. For long sequences like audio, that's a big win.

Cartesia also built streaming as a first-class concern. The first byte goes out as soon as the first input chunk arrives.

6.4 Where It's Used

If ElevenLabs is "best quality," Cartesia is "best balance of latency and quality."

Chapter 7 · Sesame (Iribe, March 2025) — "Voice Presence"

7.1 Brendan Iribe and Sesame

Brendan Iribe co-founded Oculus VR. After the Facebook acquisition and stops at Anduril, he founded Sesame in 2024 and demoed publicly in March 2025.

The Sesame concept is "voice presence" — not just natural-sounding, but a voice that makes you feel someone is there. Breath, hesitation, "uhh", backchannels ("yeah", "mhm"), interruptions handled naturally.

7.2 Why the Demo Went Viral

The 30-second demo in March 2025 spread fast on social. Why:

If ElevenLabs/Cartesia made "natural voice," Sesame made "someone is here."

7.3 What's Different Under the Hood

Sesame partially published a paper. The key ideas:

7.4 Caveats

Even so, the direction (treating voice as presence rather than as TTS output) is where ElevenLabs and Cartesia will soon follow.

Chapter 8 · ChatTTS / F5-TTS / XTTS-v2 — Open-Source TTS

8.1 ChatTTS — A Chinese Team's Natural English TTS

ChatTTS is an open-source TTS released in 2024 by a Chinese team. Notable for:

8.2 F5-TTS — Trending #1 on HuggingFace

F5-TTS dropped in late 2024 and hit #1 trending on HuggingFace. Also a hot topic in the Korean developer community.

8.3 XTTS-v2 (Coqui) — The Cloning Classic

Coqui was an active open-source TTS company in 2023–2024. The company itself shut down but XTTS-v2 weights remain on HuggingFace.

from TTS.api import TTS

tts = TTS(model_name="tts_models/multilingual/multi-dataset/xtts_v2").to("cuda")
tts.tts_to_file(
    text="Hello there.",
    speaker_wav="reference.wav",
    language="en",
    file_path="output.wav",
)

8.4 Tortoise TTS — Slow but Quality

Tortoise is a relatively old open-source TTS from 2022. Inference is very slow (minutes) but quality was good enough that it was the de facto open-source TTS for a while. Today ChatTTS and F5-TTS have taken that spot.

8.5 Open-Source TTS Cheat Sheet

ModelEnglish qualityMultilingualInference speedLicense
ChatTTSExcellentWeakFastNon-commercial concerns
F5-TTSGoodEnglish/ChineseMediumNon-commercial
XTTS-v2Good17 languagesMediumCPL (conditional commercial)
TortoiseExcellentEnglishVery slowApache 2.0

License check is mandatory for commercial use. F5-TTS has an explicit non-commercial clause and is off-limits for commercial products.

Chapter 9 · Realtime API — OpenAI / Google / ElevenLabs Conversational

9.1 What "Realtime" Means

Traditional voice pipelines are STT then LLM then TTS, in series. Each stage adds latency, distortion, and wait. Realtime APIs collapse that into one model: voice in, voice out.

Upsides:

Downsides:

9.2 OpenAI Realtime API (gpt-4o-realtime)

Launched late 2024. WebSocket-based.

const ws = new WebSocket("wss://api.openai.com/v1/realtime?model=gpt-4o-realtime-preview")
ws.on("open", () => {
  ws.send(JSON.stringify({
    type: "session.update",
    session: { voice: "alloy", instructions: "You are a helpful Korean assistant." }
  }))
})
ws.on("message", (data) => {
  const event = JSON.parse(data)
  if (event.type === "response.audio.delta") {
    // base64 PCM chunk
  }
})

GPT-4o's voice model takes audio in and produces audio out directly. Korean and Japanese supported.

9.3 Google Live API (Gemini 2)

Gemini 2's Live API. Similar WebSocket interface. Strength is integration with the Google ecosystem (Search, Maps, Calendar).

9.4 ElevenLabs Conversational v2

ElevenLabs expanded from pure TTS into a voice-agent platform. STT is in-house plus Deepgram option, LLM is your pick (OpenAI/Anthropic/Google), TTS is ElevenLabs voices. Plug-and-play.

9.5 When to Use Realtime vs Split

Realtime is good for:

Split is good for:

Chapter 10 · Voice Agents — Vapi / Retell / Bland / Synthflow

10.1 What a Voice-Agent Platform Is

To ship a call-center bot you need STT/LLM/TTS wired together, phone-network integration (SIP/Twilio), and turn-taking/interruption/call-routing logic. Platforms that do all of that exploded in 2024–2025.

10.2 Vapi

SF startup, YC-backed. Notable for:

10.3 Retell AI

Vapi's most direct competitor. Slicker UI, easier-to-read live call transcripts. Pricing is similar.

10.4 Bland AI

Sales-call specialist. Strong for high-volume outbound (e.g. real-estate cold calls). Per-call pricing is cheap.

10.5 Synthflow

EU no-code voice-agent builder. GUI-driven flow editor. Non-developers in ops can use it.

10.6 Comparison

PlatformStrengthWeaknessPer-min (USD)
VapiFlexible, good APIUI is plain0.05–0.15 + model
Retell AIClean UI, good transcriptsSimilar pricing0.07–0.15 + model
Bland AIHigh-volume outboundWeaker for inbound/complex~0.09 per call
SynthflowNo-code, EU dataAPI flexibility weaker0.13+ + model

10.7 Build vs Buy

Below 10K calls per minute, a platform is almost always cheaper. Above that — or if data can't leave the cluster — build with LiveKit + Deepgram + Cartesia.

Chapter 11 · Korea — Naver CLOVA, Kakao KOTTS, SKT NUGU

11.1 Naver CLOVA Voice / CLOVA Studio

Naver has CLOVA Voice (TTS), CLOVA Speech (STT), and HyperCLOVA X (LLM) — a full voice stack. Korean naturalness is ahead of ElevenLabs. Billed per minute or per character.

11.2 Kakao KOTTS

Kakao Enterprise's Korean TTS. Aimed at B2B (call center, announcements). Integrates with the Kakao Talk bot builder.

11.3 SKT NUGU

SK Telecom's voice assistant platform. NUGU speakers, TMAP voice navigation, NUGU Candy — strong in the consumer market.

11.4 Coway Sonatts and Others

A handful of Korean enterprises have built proprietary Korean TTS. Limited public exposure.

11.5 Korean STT — CLOVA vs Deepgram vs Whisper

ModelKorean WERStrengthWeakness
Naver CLOVA Speech~5–7%Korean domain tuning, Korean proper nounsWeak global integration
Deepgram (Korean)~9%Low latency, globalWeak domain tuning
Whisper v3 turbo~8%Free, multilingualDiarization separate
ParakeetN/A (English-centric)--

For a Korean company serving Korean users, CLOVA is the top pick. For global plus Korean, Whisper turbo or Deepgram.

11.6 Korean Voice-Agent Cases

Korean voice agents lag global by 2–3 years but are catching up fast.

Chapter 12 · Japan — VOICEVOX (Open Source), Coeiroink, GPT-SoVITS, Bert-VITS2

12.1 VOICEVOX — Japan's De Facto Open-Source TTS

VOICEVOX has overwhelming mindshare in Japan. Notable for:

12.2 Coeiroink

Similar to VOICEVOX, slightly more permissive on character licensing. Preferred by some creators.

12.3 GPT-SoVITS

Zero-shot voice cloning TTS popular in the Japanese and Chinese communities. Cloning from under one minute. De facto standard for Japanese voice content creators.

12.4 Bert-VITS2

Another open-source favorite. BERT-based text encoder + VITS decoder. Strong in Japanese and Chinese.

12.5 Japanese Commercial TTS

Commercial is dominated by the global three, but in the Japanese content market (VTubers, video, games) VOICEVOX and GPT-SoVITS dominate.

12.6 Japanese STT

ModelJapanese WERNotes
Whisper v3 turbo~8%Most popular
AssemblyAI~9%Post-processing strength
Google STT~7%Japanese domain tuning
Azure Speech~7%Many Japanese voices
Deepgram~11%Weak in Japanese

For Japanese, Deepgram is surprisingly weak; Google and Azure often win.

Chapter 13 · Who Should Choose What — Call Center / Game NPC / Audiobook / Interpretation

13.1 Inbound Call-Center Bot

Goal: fast response + natural Korean/English + interruption handling + tool calls

Recommended:

Alternative: OpenAI Realtime API alone (enough for a simple bot, more expensive)

13.2 Game NPC Voiceover

Goal: character-voice consistency + emotion + multilingual

Recommended:

13.3 Audiobook / Podcast

Goal: natural long-form pacing, emotion, accurate pronunciation

Recommended:

13.4 Live Interpretation

Goal: ultra-low-latency STT + instant translation + natural TTS

Recommended:

13.5 Video Captioning / Content Post-Processing

Goal: accuracy + diarization + chapters/summary

Recommended:

13.6 Cost-Sensitive + Private Data

Goal: data cannot leave the cluster, GPU-only operation

Recommended:

13.7 One-Line Matrix

ScenarioSTTTTSNotes
Korean call centerCLOVA SpeechCLOVA VoiceDomain tuning
English call centerDeepgramCartesia Sonic 2Low latency
Game NPC(n/a)ElevenLabs V3Emotion tags
Audiobook(n/a)ElevenLabs v2Long form
Live interpretationDeepgramCartesiaOr OpenAI Realtime
Media captioningAssemblyAI(n/a)Chapters/summary
Private on-premParakeetXTTS-v2NVIDIA Riva
Japanese contentWhisperVOICEVOXCharacter voices

Chapter 14 · Wrap-Up — The Big Picture of Voice AI in 2026

Three big currents.

First, STT is basically solved. Whisper turbo, Deepgram Nova-3, and Parakeet 1.1 have made sub-5% English WER routine. What remains is domain adaptation (medical/legal terms), multilingual accuracy (especially low-resource languages), and side info like diarization and emotion metadata.

Second, TTS is moving from "natural voice" to "voice presence." ElevenLabs and Cartesia almost finished naturalness; Sesame redefined the bar with "someone is here." In late 2026 to 2027 ElevenLabs and Cartesia will probably catch up to that territory.

Third, unified (Realtime API) is eating the split pipeline. Simple bots can ship with OpenAI Realtime API alone. The split pipeline survives where (a) brand voice matters, (b) complex tool chains are needed, (c) audio data must be separately audited.

Voice AI is no longer a fun demo. In 2026 it ships in call centers, automotive infotainment, games, education, and healthcare. Things to watch in the next 1–2 years: (1) whether Sesame can actually ship at scale, (2) whether open-source TTS narrows the ElevenLabs gap, (3) whether Whisper turbo gets another jump.

References

Read the original: https://www.youngju.dev/blog/culture/2026-05-15-voice-ai-2026-elevenlabs-cartesia-sesame-whisper-deepgram-parakeet-deep-dive.en

Comments

No comments yet.

Sign in to leave a comment