- The Two Phases of LLM Inference: Prefill and Decode
- KV Cache: The Memory Dilemma of Attention
- PagedAttention (vLLM): Virtual Memory Saves LLMs
- Continuous Batching: Maximizing Throughput
- Quantization: Trade Precision for Speed and Memory
- Speculative Decoding: Free Lunch Exists
- Tensor Parallelism and Pipeline Parallelism
- vLLM vs TGI vs TensorRT-LLM: Framework Comparison
- Production LLM Serving Stack
- vLLM Settings in This Post That Have Shifted Between Versions
- What to Measure First
- Failure Modes and the Order to Diagnose Them
- When Not to Use Any of This
- References
This post is a map of the whole LLM serving optimization landscape, covered in one pass. For each technique we look at the problem it was originally built to solve, and at what that problem actually looks like as a symptom in a live deployment.
If you want to go deeper on vLLM specifically, start with the vLLM internals series on this blog. That series works through the scheduler, preemption, prefix caching, and length limits one at a time, checking each against the official docs and the source. A few of the vLLM settings that appear in this post have shifted between versions; those are called out in a section of their own below.
The Two Phases of LLM Inference: Prefill and Decode
LLM text generation splits into two fundamentally different phases. Without understanding this split, optimization is impossible.
Phase 1: PREFILL (process the input)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Input: "What is the capital of France?"
└─ all 9 tokens processed at once
What happens:
- All input tokens are processed in parallel (big matrix multiply!)
- Q, K, V are computed for each input token
- KV cache is created (saves K, V for later reuse)
- First output token is generated
Characteristics:
- GPU operation: COMPUTE-BOUND (matrix × matrix)
- GPU utilization: HIGH ✅
- Latency metric: TTFT (Time To First Token)
Phase 2: DECODE (generate tokens one-by-one)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Generates: "Paris" → "is" → "the" → "capital" → ...
What happens:
- Generate exactly one token per forward pass
- Compute Q for the new token, attend over cached K, V
- Must read ALL model weights for every single token
Characteristics:
- GPU operation: MEMORY-BOUND (matrix × vector)
- GPU utilization: LOW (often 5–20%!)
- Throughput metric: TBT (Time Between Tokens)
This is why LLM serving is hard to optimize:
the two phases have completely different bottlenecks!
Measuring it in practice:
import torch
import time
from transformers import AutoModelForCausalLM, AutoTokenizer
def measure_llm_phases(model_name="meta-llama/Llama-3.2-1B"):
model = AutoModelForCausalLM.from_pretrained(
model_name, torch_dtype=torch.float16, device_map="cuda"
)
tokenizer = AutoTokenizer.from_pretrained(model_name)
prompt = "Explain the transformer architecture in detail:"
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
input_len = inputs["input_ids"].shape[1]
# Measure prefill time
torch.cuda.synchronize()
t0 = time.perf_counter()
with torch.no_grad():
_ = model(**inputs) # forward pass on input only
torch.cuda.synchronize()
t_prefill = time.perf_counter() - t0
# Measure decode time
t0 = time.perf_counter()
with torch.no_grad():
generated = model.generate(
inputs["input_ids"],
max_new_tokens=50,
do_sample=False
)
torch.cuda.synchronize()
t_total = time.perf_counter() - t0
t_decode = t_total - t_prefill
n_new = generated.shape[1] - input_len
print(f"Input tokens: {input_len}")
print(f"Prefill time (TTFT): {t_prefill*1000:.1f}ms")
print(f"Generated tokens: {n_new}")
print(f"Decode time: {t_decode*1000:.1f}ms")
print(f"Per-token decode time: {t_decode/n_new*1000:.1f}ms/token")
# Llama-1B on H100 (~):
# Prefill: ~5ms (linear in input length)
# Decode: ~3ms/token (proportional to model size, batch-dependent)
What matters in this measurement is not the absolute numbers but the difference in character between the two phases. Run the code above a few times, varying only the input length, and prefill time grows roughly linearly with the input token count while per-token decode time barely moves. Prefill multiplies matrices against matrices, which saturates the GPU's compute units; decode reads the entire set of model weights out of memory once just to produce a single token.
That is where the leverage of batching comes from. In the decode phase, the cost of reading the weights is the same no matter how many requests are in the batch. Fetching the weights once and serving one request moves exactly as many bytes as serving 64. So growing the batch divides that cost almost cleanly across requests. As the batch grows, decode drifts from memory-bound toward compute-bound, and once it arrives there, adding more requests stops buying throughput.
The catch is that what caps the batch is not the GPU's compute capacity but KV cache memory. Every technique below follows from that one sentence.
KV Cache: The Memory Dilemma of Attention
What Happens Without a KV Cache?
Autoregressive generation WITHOUT KV cache:
Step 1: [token_1] → generate token_2
- Compute Q1,K1,V1 for token_1
- Compute Q2,K2,V2 for token_2 (partial)
- Attention: Q2 × [K1,K2]^T
- Ops: 2^2 = 4 dot products
Step 2: [token_1, token_2] → generate token_3
- Re-compute K1,V1 (wasted work!)
- Re-compute K2,V2 (wasted work!)
- Compute Q3,K3,V3
- Attention: Q3 × [K1,K2,K3]^T
- Ops: 3^2 = 9 dot products
Step N: O(N^2) operations per token
Total for L tokens: O(L^3) total compute
100 tokens: 1,000,000 dot products
1000 tokens: 1,000,000,000 dot products
KV Cache: Reuse Previous Computation
import torch
import torch.nn as nn
import math
class AttentionWithKVCache(nn.Module):
def __init__(self, d_model, n_heads):
super().__init__()
self.n_heads = n_heads
self.d_k = d_model // n_heads
self.W_q = nn.Linear(d_model, d_model, bias=False)
self.W_k = nn.Linear(d_model, d_model, bias=False)
self.W_v = nn.Linear(d_model, d_model, bias=False)
self.W_o = nn.Linear(d_model, d_model, bias=False)
# KV cache storage
self.k_cache = None # (batch, heads, past_len, d_k)
self.v_cache = None
def forward(self, x, use_cache=True):
batch, seq, d = x.shape
Q = self.W_q(x).view(batch, seq, self.n_heads, self.d_k).transpose(1,2)
K = self.W_k(x).view(batch, seq, self.n_heads, self.d_k).transpose(1,2)
V = self.W_v(x).view(batch, seq, self.n_heads, self.d_k).transpose(1,2)
if use_cache and self.k_cache is not None:
# Append new K, V to the cache
K = torch.cat([self.k_cache, K], dim=2)
V = torch.cat([self.v_cache, V], dim=2)
if use_cache:
self.k_cache = K.detach()
self.v_cache = V.detach()
# Q is only the current token(s); K, V are the full sequence
scale = math.sqrt(self.d_k)
scores = torch.matmul(Q, K.transpose(-2,-1)) / scale
weights = torch.softmax(scores, dim=-1)
output = torch.matmul(weights, V)
return output.transpose(1,2).contiguous().view(batch, seq, d)
def compute_kv_cache_bytes(seq_len, n_layers, n_kv_heads, head_dim,
batch_size, dtype_bytes=2):
"""
KV cache memory in bytes.
Factor of 2: one tensor for K, one for V.
"""
return 2 * n_layers * n_kv_heads * head_dim * seq_len * batch_size * dtype_bytes
# Llama 3.1 70B (uses GQA: 8 KV heads, 64 Q heads):
size = compute_kv_cache_bytes(
seq_len=4096, n_layers=80, n_kv_heads=8,
head_dim=128, batch_size=1, dtype_bytes=2
)
print(f"KV cache (Llama-70B, seq=4096, batch=1): {size/1e9:.1f} GB")
# Result: ~6.7 GB per request
# batch=32: ~214 GB → won't fit in one H100 (80 GB)!
The single most consequential argument in that formula, in practice, is n_kv_heads. The example passes 8 because Llama 3.1 70B uses GQA (Grouped Query Attention): 64 query heads, but only 8 key and value heads, with each KV head shared by 8 query heads. Had the model kept a separate K and V per head the old way, the same formula would take 64, and the KV cache would come out at 54 GB per request instead of 6.7 GB. This is why almost every recent model adopts GQA or MQA. The memory budget at serving time is the number of requests you can hold concurrently, and that number is your unit cost.
When planning a deployment it is more useful to run the formula backwards. Start with total GPU memory, subtract what the model weights take, and what remains is the capacity available for KV cache. Divide that by the per-request cache size from the formula above and you get an upper bound on how many requests can be alive at once. If that bound is below your target concurrency, no amount of scheduler tuning will reach the target — the only moves left are quantizing the weights, cutting the maximum length, or adding GPUs. Put the other way around: one division tells you whether you have a tuning problem or a hardware problem.
One more thing. Cache size is exactly proportional to sequence length. If a service leaves the maximum length open at 32768 while the median real request is 1200 tokens, worst-case capacity planning reserves more than twenty times what is actually needed. That is why trimming the maximum length to fit the real workload is usually the cheapest optimization on the table.
The Memory Fragmentation Problem
Traditional KV cache allocation (pre-allocate max_seq_len per request):
┌───────────────────────────────────────────────────────┐
│ Request 1: current_len=100, reserved=512 │
│ ████████████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ │
│ [used: 100] [wasted: 412 slots = 80%!] │
├───────────────────────────────────────────────────────┤
│ Request 2: current_len=50, reserved=512 │
│ ██████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ │
│ [used: 50] [wasted: 462 slots = 90%!] │
├───────────────────────────────────────────────────────┤
│ Request 3: current_len=300, reserved=512 │
│ █████████████████████████████████████░░░░░░░░ │
│ [used: 300] [wasted: 212 slots = 41%!] │
└───────────────────────────────────────────────────────┘
Total allocated: 3 × 512 = 1536 slots
Total used: 450 slots
Wasted: 1086 slots = 71%
Internal fragmentation (allocated but unused) +
External fragmentation (gaps between allocations)
→ Typical real-world GPU memory utilization: 20–40%
PagedAttention (vLLM): Virtual Memory Saves LLMs
The Core Insight: OS Virtual Memory Applied to KV Cache
Kwon et al. (UC Berkeley, 2023): "Operating systems solved memory fragmentation decades ago. Apply the same idea to KV cache."
The OS Virtual Memory Lesson:
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
virtual address → page table → physical address
Process sees contiguous virtual space
Physical memory can be non-contiguous
→ No fragmentation, efficient RAM usage
PagedAttention Analogy:
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
virtual KV slot → block table → physical block
Sequence sees contiguous virtual slots
Physical GPU blocks can be non-contiguous
→ Near-zero fragmentation, efficient GPU VRAM usage
PagedAttention Memory Layout:
GPU memory split into fixed-size blocks (default: 16 tokens each):
┌─────────────────────────────────────────────────────────┐
│ Physical KV Cache Blocks │
│ Block 0: [tok0–15] Block 1: [tok16–31] │
│ Block 2: [tok32–47] Block 3: [tok48–63] │
│ Block 4: [tok64–79] Block 5: FREE │
│ Block 6: FREE Block 7: FREE │
└─────────────────────────────────────────────────────────┘
Block Table (same role as OS page table):
┌──────────┬──────────────────────────────────────────────┐
│ Request │ Virtual block → Physical block mapping │
├──────────┼──────────────────────────────────────────────┤
│ Req 1 │ virt[0]→phys[0], virt[1]→phys[2] │
│ │ (tokens 0–15: block 0; tokens 32–47: block 2) │
├──────────┼──────────────────────────────────────────────┤
│ Req 2 │ virt[0]→phys[1], virt[1]→phys[3] │
│ │ (tokens 0–15: block 1; tokens 16–31: block 3) │
└──────────┴──────────────────────────────────────────────┘
Key properties:
- Blocks are allocated ON-DEMAND as the sequence grows
- Internal fragmentation < 1 block = at most 15 wasted slots
- Blocks can be SHARED across requests (same prefix)!
# vLLM with PagedAttention:
from vllm import LLM, SamplingParams
import time
def benchmark_vllm():
llm = LLM(
model="meta-llama/Llama-3.2-8B-Instruct",
gpu_memory_utilization=0.9,
max_model_len=8192,
block_size=16,
max_num_seqs=256,
)
prompts = [
"Short question: What is Python?",
"Medium question: " + "Explain the history of machine learning. " * 5,
"Long question: " + "How do you build a transformer from scratch? " * 10,
] * 20 # 60 requests of varying lengths
params = SamplingParams(temperature=0.0, max_tokens=100)
t0 = time.perf_counter()
outputs = llm.generate(prompts, params)
elapsed = time.perf_counter() - t0
total_tokens = sum(len(o.outputs[0].token_ids) for o in outputs)
print(f"Requests: {len(prompts)}")
print(f"Tokens generated: {total_tokens}")
print(f"Elapsed: {elapsed:.1f}s")
print(f"Throughput: {total_tokens/elapsed:.0f} tokens/s")
# Memory efficiency improvement:
# Traditional: 20–40% of GPU memory used for actual KV cache
# PagedAttention: >95% of GPU memory used for actual KV cache
# Result: 2–3× more concurrent requests on the same GPU
Prefix Caching: Share Common Prompts
# Enable prefix caching in vLLM:
llm = LLM(
model="meta-llama/Llama-3.2-8B-Instruct",
enable_prefix_caching=True,
)
# Many requests sharing a long system prompt:
system = "You are an expert software engineer. " * 100 # long system prompt
requests = [
system + "User: What is a binary search tree?",
system + "User: How does garbage collection work?",
system + "User: Explain ACID properties in databases.",
]
# The KV cache for `system` is computed ONCE and shared across all 3 requests.
# Prefill cost: computed 1 time instead of 3 times (3× savings on prefill!)
# This matters hugely for RAG pipelines where context is repeated.
Continuous Batching: Maximizing Throughput
The Problem with Static Batching
Static (request-level) batching:
GPU batch at each step:
Step 1: [Req1: running] [Req2: running] [Req3: running]
Step 2: [Req1: running] [Req2: DONE ] [Req3: running]
Step 3: [Req1: running] [ idle/wait ] [Req3: running] ← GPU waste!
Step 4: [Req1: DONE ] [ idle/wait ] [Req3: running] ← GPU waste!
Step 5: [ idle/wait ] [ idle/wait ] [Req3: DONE ] ← GPU waste!
New requests must wait until the ENTIRE batch finishes.
GPU waste rate: often 50%+
Continuous Batching: Dynamic Scheduling Every Token Step
Continuous (iteration-level) batching:
Step 1: [Req1] [Req2] [Req3]
Step 2: [Req1] [Req2] [Req3]
Step 3: [Req1] [Req4] [Req3] ← Req2 done → Req4 inserted immediately!
Step 4: [Req5] [Req4] [Req3] ← Req1 done → Req5 inserted!
Step 5: [Req5] [Req4] [Req6] ← Req3 done → Req6 inserted!
GPU is at maximum utilization at every step.
Throughput improvement over static batching: 2–4×
from vllm.engine.async_llm_engine import AsyncLLMEngine
from vllm.engine.arg_utils import AsyncEngineArgs
import asyncio
async def run_continuous_batching_server():
engine_args = AsyncEngineArgs(
model="meta-llama/Llama-3.2-8B-Instruct",
max_num_seqs=256, # max concurrent sequences
max_num_batched_tokens=8192, # max tokens per batch step
)
engine = AsyncLLMEngine.from_engine_args(engine_args)
async def generate_one(prompt, req_id):
from vllm import SamplingParams
params = SamplingParams(temperature=0.7, max_tokens=200)
async for output in engine.generate(prompt, params, request_id=req_id):
if output.finished:
return output.outputs[0].text
# Requests submitted concurrently — engine handles continuous batching
results = await asyncio.gather(
generate_one("Explain quantum entanglement.", "r1"),
generate_one("Write a Python quicksort.", "r2"),
generate_one("Summarize the French Revolution.", "r3"),
)
for r in results:
print(r[:80])
Quantization: Trade Precision for Speed and Memory
Why Quantization?
LLM memory footprint (FP16):
Llama 3.1 8B: 16 GB
Llama 3.1 70B: 140 GB
Llama 3.1 405B: 810 GB
Common GPU memory:
RTX 4090: 24 GB → tight even for 8B
A100 80GB: → 70B impossible on one card
H100 80GB: → 70B impossible on one card
H100 ×8 (640 GB): → 70B fine, 405B barely
Memory savings with quantization:
FP16 (16-bit): baseline
INT8 (8-bit): 50% saved, ~1% accuracy loss
INT4 (4-bit): 75% saved, ~2–3% accuracy loss
INT3 (3-bit): 81% saved, use cautiously
INT2 (2-bit): 88% saved, usually unacceptable
Post-Training Quantization: INT8 (LLM.int8())
from transformers import AutoModelForCausalLM, BitsAndBytesConfig
import torch
# INT8 quantization — Dettmers et al., 2022 (bitsandbytes):
config_int8 = BitsAndBytesConfig(
load_in_8bit=True,
# Optionally keep certain layers in FP16 (e.g., output head)
llm_int8_skip_modules=["lm_head"],
)
model_int8 = AutoModelForCausalLM.from_pretrained(
"meta-llama/Llama-3.1-70B-Instruct",
quantization_config=config_int8,
device_map="auto",
)
# 70B model: 140 GB (FP16) → 70 GB (INT8), ~1% accuracy loss
# The key insight behind LLM.int8():
# Problem: activation outliers in certain channels ruin naive INT8 quality
# Solution: "Mixed-precision decomposition"
# - Detect outlier channels (top ~1% by magnitude)
# - Keep those channels in FP16
# - Quantize all other channels to INT8
# → Near-lossless quality with ~50% memory savings
4-bit Quantization: NF4 and GPTQ
# NF4 quantization (QLoRA paper, Dettmers et al. 2023):
config_4bit = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_compute_dtype=torch.float16, # compute in FP16
bnb_4bit_quant_type="nf4", # NormalFloat4
bnb_4bit_use_double_quant=True, # quantize the scale too!
)
model_4bit = AutoModelForCausalLM.from_pretrained(
"meta-llama/Llama-3.1-70B-Instruct",
quantization_config=config_4bit,
device_map="auto",
)
# 70B: 140 GB → 35 GB, ~2–3% accuracy loss
# Why NF4?
# Neural network weights are approximately normally distributed.
# NF4 uses 16 codepoints placed at equal-probability quantiles
# of a standard normal distribution.
# Each codepoint covers an equal probability mass → minimal quantization error
# vs. uniform INT4 which distributes points evenly on the number line.
# Double quantization:
# Quantization scale factors are FP32: 1 per group of 64 weights
# Double-quant quantizes those scale factors to INT8 too
# Net savings: ~0.5 additional bits per weight
# GPTQ (Frantar et al., 2022) — layer-wise optimal quantization:
from auto_gptq import AutoGPTQForCausalLM
model_gptq = AutoGPTQForCausalLM.from_quantized(
"TheBloke/Llama-2-70B-GPTQ",
device="cuda:0",
use_triton=True, # Triton kernels for faster inference
)
# GPTQ uses the Hessian of each layer's loss to minimize quantization error.
# Generally highest accuracy among INT4 methods.
AWQ: Activation-Aware Weight Quantization
# AWQ (Lin et al., 2023) key insight:
# Not all weights are equally important!
# ~1% of channels produce large activations — these are "salient"
# Naively quantizing them to INT4 crushes quality
# AWQ solution:
# 1. Run calibration data, record activation magnitudes per channel
# 2. Scale up salient channels in the weight matrix (per-channel scaling)
# 3. Quantize everything to INT4 — the scaling absorbs the error for salient channels
from awq import AutoAWQForCausalLM
from transformers import AutoTokenizer
model_path = "meta-llama/Llama-3.1-8B-Instruct"
quant_path = "llama-3.1-8b-awq"
model = AutoAWQForCausalLM.from_pretrained(model_path)
tokenizer = AutoTokenizer.from_pretrained(model_path)
quant_config = {
"zero_point": True,
"q_group_size": 128,
"w_bit": 4,
"version": "GEMM"
}
model.quantize(tokenizer, quant_config=quant_config)
model.save_quantized(quant_path)
# AWQ vs GPTQ:
# AWQ: faster inference (hand-tuned CUDA/Triton kernels)
# ~25% of FP16 memory
# accuracy: slightly below GPTQ
# GPTQ: higher accuracy (Hessian-based error minimization)
# similar inference speed
# same memory as AWQ
Quantization Comparison Table
Llama 3.1 70B quantization comparison (single H100):
┌──────────┬──────────┬────────────┬──────────┬───────────────────┐
│ Format │ Memory │ Throughput │ MMLU │ Hardware needed │
├──────────┼──────────┼────────────┼──────────┼───────────────────┤
│ FP16 │ 140 GB │ baseline │ 80.9% │ 8× H100 │
│ BF16 │ 140 GB │ +5% │ 80.9% │ 8× H100 │
│ INT8 │ 70 GB │ +10% │ 80.2% │ 2× H100 │
│ GPTQ-4b │ 36 GB │ +30% │ 79.8% │ 1× H100 │
│ AWQ-4b │ 36 GB │ +35% │ 79.5% │ 1× H100 │
│ GGUF-Q4 │ 38 GB │ CPU ok │ 79.1% │ CPU or 1× H100 │
└──────────┴──────────┴────────────┴──────────┴───────────────────┘
Speculative Decoding: Free Lunch Exists
The Idea: Draft Fast, Verify in Parallel
Standard decode:
70B model generates 1 token = 1 forward pass = ~10ms
100 tokens = ~1000ms = 1 second
Speculative decoding (Leviathan et al., 2023):
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Step 1: Draft model (7B) generates K tokens quickly
["Paris"] ["is"] ["the"] ["capital"]
4 tokens in ~2ms (7B model)
Step 2: Target model (70B) verifies all K tokens in ONE forward pass!
Process all 4 draft tokens in parallel → ~10ms
(same cost as generating just 1 token normally)
Step 3: Verify each draft token:
"Paris" ✅ "is" ✅ "the" ✅ "capital" ❌
→ accept 3 tokens, reject from position 4
Step 4: Resample from target model distribution at rejection point
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Result: 3 accepted tokens in ~12ms (vs 30ms standard decode)
Speedup: 2.5× (varies with acceptance rate ~70–90%)
Quality: ZERO degradation (target model is the arbiter)
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
def speculative_decode(
target_model,
draft_model,
input_ids,
max_new_tokens=100,
K=4, # number of draft tokens per speculation
temperature=1.0,
):
"""
Speculative decoding: draft model proposes K tokens,
target model verifies them all in one forward pass.
Guarantees exactly the same distribution as target-only decoding.
"""
generated = input_ids.clone()
while generated.shape[1] < input_ids.shape[1] + max_new_tokens:
# --- Phase 1: Draft model generates K candidates ---
draft_ids = []
draft_probs = []
ctx = generated.clone()
for _ in range(K):
with torch.no_grad():
out = draft_model(ctx)
logits = out.logits[:, -1, :] / max(temperature, 1e-5)
probs = torch.softmax(logits, dim=-1)
tok = torch.multinomial(probs, 1)
draft_ids.append(tok)
draft_probs.append(probs[0, tok[0, 0]])
ctx = torch.cat([ctx, tok], dim=1)
# --- Phase 2: Target model verifies K positions simultaneously ---
candidate = torch.cat([generated] + draft_ids, dim=1)
with torch.no_grad():
tgt_out = target_model(candidate)
# logits for positions where draft tokens are placed
tgt_logits = tgt_out.logits[:, len(generated[0])-1:-1, :]
tgt_probs = torch.softmax(tgt_logits / max(temperature, 1e-5), dim=-1)
# --- Phase 3: Accept/reject each draft token ---
n_accepted = 0
for i in range(K):
token_id = draft_ids[i][0, 0].item()
p_target = tgt_probs[0, i, token_id].item()
p_draft = draft_probs[i].item()
# Acceptance probability: min(1, p_target / p_draft)
accept_p = min(1.0, p_target / max(p_draft, 1e-8))
if torch.rand(1).item() < accept_p:
generated = torch.cat([generated, draft_ids[i]], dim=1)
n_accepted += 1
else:
# Reject: resample from adjusted target distribution
adjusted = torch.clamp(tgt_probs[0, i] - tgt_probs[0, i], min=0)
# Correct adjusted distribution (residual of target minus draft)
diff = tgt_probs[0, i].clone()
diff[token_id] = max(0.0, diff[token_id] - p_draft)
diff = diff / diff.sum().clamp(min=1e-8)
new_tok = torch.multinomial(diff.unsqueeze(0), 1)
generated = torch.cat([generated, new_tok], dim=1)
break
if n_accepted == K:
# All accepted: also take the target model's bonus token
bonus_logits = tgt_out.logits[:, -1, :] / max(temperature, 1e-5)
bonus_probs = torch.softmax(bonus_logits, dim=-1)
bonus_tok = torch.multinomial(bonus_probs, 1)
generated = torch.cat([generated, bonus_tok], dim=1)
return generated
# Real speedups observed (A100, Llama-2 70B + Llama-2 7B draft):
# K=4: 2.3× speedup, acceptance rate ~80%
# K=8: 2.7× speedup, acceptance rate ~75%
# Optimal K depends on draft/target quality ratio
Tensor Parallelism and Pipeline Parallelism
Tensor Parallelism: Split Layers Across GPUs
Tensor Parallelism (Shoeybi et al., 2019 — Megatron-LM):
70B model, 8 GPUs, 64 attention heads:
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
GPU 0: heads 0–7 (W_q slice: 1/8 of full matrix)
GPU 1: heads 8–15
GPU 2: heads 16–23
GPU 3: heads 24–31
GPU 4: heads 32–39
GPU 5: heads 40–47
GPU 6: heads 48–55
GPU 7: heads 56–63
Each GPU computes its heads independently,
then All-Reduce merges results.
Communication cost:
1 All-Reduce per attention layer
1 All-Reduce per FFN layer
NVLink (H100): 900 GB/s bidirectional → viable
PCIe: 64 GB/s → too slow for TP>2
import torch
import torch.distributed as dist
def tensor_parallel_linear(x, W_local, rank, world_size):
"""
Column-parallel linear (W split along output dimension).
x: (batch, seq, d_model) -- replicated on all GPUs
W_local: (d_model, d_out//world_size) -- each GPU holds a shard
"""
# Each GPU computes its output shard
out_local = x @ W_local # (batch, seq, d_out//world_size)
# All-Gather to reconstruct full output on every GPU
out_list = [torch.zeros_like(out_local) for _ in range(world_size)]
dist.all_gather(out_list, out_local)
out_full = torch.cat(out_list, dim=-1) # (batch, seq, d_out)
return out_full
# For row-parallel (W split along input dimension):
def tensor_parallel_linear_row(x_local, W_local, rank, world_size):
"""
Row-parallel linear: x is already sharded across GPUs.
x_local: (batch, seq, d_in//world_size)
W_local: (d_in//world_size, d_out)
"""
partial = x_local @ W_local # (batch, seq, d_out) — partial sum
dist.all_reduce(partial, op=dist.ReduceOp.SUM) # sum partial results
return partial
Pipeline Parallelism: Split Layers Sequentially
Pipeline Parallelism:
70B model, 80 layers, 4 GPUs:
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
GPU 0: layers 0–19 (embedding + first 20 transformer layers)
GPU 1: layers 20–39
GPU 2: layers 40–59
GPU 3: layers 60–79 + LM head
With micro-batching to hide pipeline bubbles:
| mb1 | mb2 | mb3 | mb4 | mb5 |
GPU 0 →→→ [f1 ] [f2 ] [f3 ] [f4 ] [f5 ] [b5 ] [b4 ] [b3 ] [b2 ] [b1 ]
GPU 1 [ ] [f1 ] [f2 ] [f3 ] [f4 ] [f5 ] [b5 ] [b4 ] [b3 ] [b2 ] [b1 ]
GPU 2 [ ] [f1 ] [f2 ] [f3 ] [f4 ] [f5 ] [b1 ]
GPU 3 [ ] [f1 ] [f2 ] [f3 ] [f4 ] [f5 ] [b5 ]
Pipeline bubble ratio = (p - 1) / (m + p - 1)
p = number of pipeline stages
m = number of micro-batches
→ Larger m = smaller bubble = better efficiency
vLLM vs TGI vs TensorRT-LLM: Framework Comparison
LLM serving framework comparison (as of early 2026):
┌───────────────────┬────────────────────────────────────────────────┐
│ Framework │ vLLM │
├───────────────────┼────────────────────────────────────────────────┤
│ Developer │ UC Berkeley / vLLM community │
│ Key innovations │ PagedAttention, continuous batching │
│ Quantization │ AWQ, GPTQ, INT8, FP8 │
│ Throughput │ ★★★★☆ High │
│ TTFT latency │ ★★★☆☆ Medium │
│ Ease of use │ ★★★★★ Very easy (Python-native) │
│ Customizability │ ★★★☆☆ Medium │
│ License │ Apache 2.0 │
│ Notes │ Most active OSS community, OpenAI-compat API │
└───────────────────┴────────────────────────────────────────────────┘
┌───────────────────┬────────────────────────────────────────────────┐
│ Framework │ TGI (Text Generation Inference) │
├───────────────────┼────────────────────────────────────────────────┤
│ Developer │ Hugging Face │
│ Key innovations │ Continuous batching, FlashAttention │
│ Quantization │ GPTQ, AWQ, bitsandbytes │
│ Throughput │ ★★★☆☆ Medium │
│ TTFT latency │ ★★★☆☆ Medium │
│ Ease of use │ ★★★★☆ Easy (Docker-first) │
│ Customizability │ ★★★★☆ High │
│ License │ HFOIL (check commercial terms) │
│ Notes │ Native HF ecosystem integration │
└───────────────────┴────────────────────────────────────────────────┘
┌───────────────────┬────────────────────────────────────────────────┐
│ Framework │ TensorRT-LLM │
├───────────────────┼────────────────────────────────────────────────┤
│ Developer │ NVIDIA │
│ Key innovations │ TensorRT graph optimization, in-flight batching│
│ Quantization │ INT8, INT4, FP8, SmoothQuant, AWQ │
│ Throughput │ ★★★★★ Highest (NVIDIA GPUs only) │
│ TTFT latency │ ★★★★★ Lowest │
│ Ease of use │ ★★☆☆☆ Complex (C++ heavy) │
│ Customizability │ ★★☆☆☆ Difficult │
│ License │ Apache 2.0 │
│ Notes │ Best raw performance; use via Triton Server │
└───────────────────┴────────────────────────────────────────────────┘
┌───────────────────┬────────────────────────────────────────────────┐
│ Framework │ llama.cpp / Ollama │
├───────────────────┼────────────────────────────────────────────────┤
│ Developer │ ggerganov / Ollama Inc. │
│ Key innovations │ GGUF quantization, CPU+GPU hybrid │
│ Quantization │ Q2–Q8 (GGUF format) │
│ Throughput │ ★★☆☆☆ Low (on CPU) │
│ TTFT latency │ ★★☆☆☆ High │
│ Ease of use │ ★★★★★ Simplest possible │
│ Customizability │ ★★☆☆☆ Limited │
│ License │ MIT │
│ Notes │ Ideal for local dev, CPU inference, demos │
└───────────────────┴────────────────────────────────────────────────┘
Decision Guide
Choose vLLM if:
- Production serving, Python team, open source preferred
- Need OpenAI-compatible API drop-in replacement
- Want the best community support and newest features fastest
Choose TGI if:
- Deep HuggingFace ecosystem integration
- Docker-first deployment culture
- Need robust SSE streaming out-of-the-box
Choose TensorRT-LLM if:
- Maximum raw performance on NVIDIA hardware
- Have a team comfortable with C++/CUDA tooling
- Enterprise production with dedicated MLOps
Choose Ollama / llama.cpp if:
- Local development, prototyping
- CPU inference required
- Simplicity over performance
Production LLM Serving Stack
Production LLM serving architecture:
Clients
│
▼
Load Balancer (Nginx / AWS ALB / Cloudflare)
│
▼
API Gateway (FastAPI / Kong)
│ Rate limiting, auth, logging, request validation
▼
Router (model selection, priority queue)
│
├──→ vLLM server A: 8B model (fast/cheap requests)
│
├──→ vLLM server B: 70B model (high-quality requests)
│
└──→ vLLM server C: domain-specific fine-tune
│
▼
Observability (Prometheus + Grafana)
Key metrics:
- TTFT p50/p95/p99
- TBT p50/p95/p99
- Throughput (tokens/s)
- GPU utilization %
- KV cache utilization %
- Request queue depth
# Production vLLM server launch command:
import subprocess
cmd = [
"python", "-m", "vllm.entrypoints.openai.api_server",
"--model", "meta-llama/Llama-3.1-8B-Instruct",
"--tensor-parallel-size", "2", # 2-GPU tensor parallelism
"--gpu-memory-utilization", "0.9",
"--max-model-len", "8192",
"--max-num-seqs", "256",
"--max-num-batched-tokens", "8192",
"--quantization", "awq",
"--enable-prefix-caching",
"--block-size", "16",
"--port", "8000",
"--disable-log-requests", # reduce logging overhead
]
# Calling the server from a client:
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="token")
response = client.chat.completions.create(
model="meta-llama/Llama-3.1-8B-Instruct",
messages=[
{"role": "system", "content": "You are a helpful AI assistant."},
{"role": "user", "content": "What is the GIL in Python?"},
],
temperature=0.0,
max_tokens=500,
stream=True, # streaming response
)
for chunk in response:
if chunk.choices[0].delta.content:
print(chunk.choices[0].delta.content, end="", flush=True)
vLLM Settings in This Post That Have Shifted Between Versions
A few of the vLLM settings written into the example code above are widely quoted numbers that no longer match the declared defaults in current source. Worth flagging before you copy them into a real deployment. The values below were checked directly against the vLLM official docs and the main-branch source on 2026-08-12, on the same basis as the vLLM deployment tuning post on this blog.
Most writing puts the default for gpu_memory_utilization at 0.9, but the declared default in CacheConfig at the time of the check was 0.92. A gap of 0.02 sounds trivial until you convert it: on an 80 GB GPU it is 1.6 GB, which in KV cache terms is several concurrent requests. When you plan capacity, use the value you verified in the version you are actually running, not a number you read somewhere.
The example above turns enable_prefix_caching on explicitly. As of the same check the declared default is already True, so the feature is on whether or not you ask for it. There is no harm in being explicit, but a claim of the form "we enabled this and it got faster" may simply not hold on recent versions.
For max_num_batched_tokens and max_num_seqs the example passes 8192 and 256, while the class-declared defaults in SchedulerConfig on main are 2048 and 128. These values are adjusted for the execution context and the environment before they are applied, so the declared value and the applied value can differ. Getting into the habit of reading them off the startup log is the safe move. What each of the two actually limits is laid out in the context window and length limits post.
The example pins block_size to 16. Which values are usable depends on the hardware and the attention backend, so rather than assuming a default, check the startup log and the engine-argument docs for the version you are on.
What to Measure First
Working down the list of techniques in this post from the top is, for most deployments, a waste of time. It is far faster to find out where this particular deployment is bottlenecked and then touch only the one thing that corresponds to it.
The vLLM OpenAI-compatible server exports Prometheus-format metrics at /metrics. With the server running, one line puts the relevant ones in front of you.
curl -s http://localhost:8000/metrics | grep -E 'vllm:(kv_cache_usage_perc|num_preemptions|prefix_cache)'
The metric names and their meanings below are the ones confirmed in the official docs.
vllm:kv_cache_usage_perc is KV cache utilization, where 1 means 100 percent. If it sits around 0.2 under normal load, GPU memory is going spare while something else keeps requests out — most likely max_num_seqs is binding first. If it is pinned near 0.95, the cache is the limit, and quantization or a shorter maximum length is what to look at.
vllm:num_preemptions is the cumulative count of requests the engine has preempted. If it keeps climbing, the engine is repeatedly throwing away KV cache it already built and recomputing it later. That is the classic reason throughput stops improving past a certain point; the diagnosis and the order to respond in are written up separately in the scheduler and preemption post.
vllm:prefix_cache_hits and vllm:prefix_cache_queries are the number of tokens the prefix cache hit on and the number it was queried for. The ratio between them is the hit rate. If your service shares a system prompt and this ratio is low, the likely cause is a value that changes per request sitting near the front of the prompt. Put a timestamp or a user name at the head of the system prompt and everything after it misses.
On the latency side, read vllm:time_to_first_token_seconds, vllm:request_time_per_output_token_seconds, and vllm:request_queue_time_seconds together. If the last one is large, the engine is not slow — the request sat in the queue — so what needs adjusting is concurrency settings and instance count, not the model or the kernels. When a complaint about sluggishness arrives, which of these three is bad completely changes where you go looking.
To generate load yourself, use the benchmark CLI that ships with vLLM.
# Run in a separate shell with the server already up
vllm bench serve \
--model meta-llama/Llama-3.1-8B-Instruct \
--host localhost --port 8000 \
--random-input-len 1024 \
--random-output-len 128 \
--num-prompts 200
vllm bench has three subcommands: serve, latency, and throughput. serve is the one that pushes real requests at a running server; the other two measure the engine with no server involved. The flags written here are only the ones confirmed in the official CLI docs. Extra options like dataset selection or request arrival rate have been renamed across versions, so check vllm bench serve --help.
One common measurement mistake is worth calling out. Synthetic load with fixed input and output lengths is far easier for the scheduler to handle than real traffic. The preemption and queueing that show up under genuinely variable-length traffic do not reproduce well under synthetic load. Use synthetic benchmark numbers only for relative comparisons between configuration A and configuration B; for absolute capacity planning, measure again against the real traffic distribution.
Failure Modes and the Order to Diagnose Them
It starts up fine but rejects requests at the maximum length. A 128k context window on the model does not mean a 128k request will go through. If the number of tokens that fit in the KV cache is smaller than max_model_len, vLLM tells you at startup and stops. It is one of the rare cases where the error message also hands you the fix: raise gpu_memory_utilization or lower max_model_len. In practice the latter comes first, because the length is usually opened wider than anything actually needs.
Throughput collapses suddenly at some point. As you raise concurrency, throughput sometimes does not gently saturate but bends over and drops past a specific point. Suspect preemption and check vllm:num_preemptions. If it is climbing, the response is counterintuitive: lower max_num_seqs. Cutting concurrency sounds like it should cut throughput, but a state of repeated preemption is already burning capacity on recomputation, so net throughput often goes up.
Quantization made it slower. Memory dropped but token generation speed did not improve. Quantization reduces how many bytes of weights have to be read, but it adds the cost of converting back to the compute precision right before the math. When the batch is small and you are in the memory-bound regime the win is large; when the batch is big enough that you are already compute-bound it can invert. Whether the kernel is optimized for your specific hardware splits the outcome too. Always re-measure at your actual batch size before deciding to change quantization formats.
Speculative decoding made it slower. The gain from this technique rests entirely on the draft model's acceptance rate. When acceptance is low you have only added the cost of drafting and thrown the result away. Acceptance drops when the draft model comes from a different family than the target, when the domain is unusual, or when you sample at a high temperature. And on a server with large batches the gain shrinks on its own: the technique exploits spare compute, and a large batch is already using it.
Prefix caching is not working. If the hit rate is lower than expected, start by checking whether the front of the prompt contains anything that changes per request. Prefix caching reuses exactly the span that matches from the beginning. Put the current time on the first line and everything after it is recomputed. Simply pushing the variable parts toward the end of the prompt can move the hit rate dramatically.
When Not to Use Any of This
Most of the techniques in this post assume online serving with enough concurrent requests to fill a batch. Break that assumption and the gains go with it.
For an internal tool with sparse traffic, neither continuous batching nor PagedAttention will be noticeable, because there are no requests to put in the batch in the first place. What actually matters for that workload is not throughput but cold start and the GPU cost of idle time. If one always-on GPU is handling a few hundred requests a day, moving to an API-provided model often wins on total cost against any amount of serving-stack tuning.
For a batch job that runs once overnight, latency metrics are meaningless. Pushing everything through the offline inference interface in one go is simpler and faster than standing up a server. Tuning settings to cut TTFT here is improving the responsiveness of a job nobody is waiting on.
If quality has not been validated yet, hold off on quantization. Quantization is a trade — you give up a little quality to buy capacity — and making that trade before you know your baseline quality means that when a quality problem shows up later, you cannot tell whether the cause is the model, the quantization, or the prompt. The order is always: establish the quality baseline first.
Finally, switching frameworks is usually the last resort. The comparison table above shows real differences between frameworks, but most of the bad numbers in real deployments come from length settings, concurrency settings, and prompt structure rather than from the choice of framework. Fix everything fixable inside the framework you already have; moving after that is not too late.
References
- vLLM Production Metrics (docs.vllm.ai) — where the names and meanings of
vllm:kv_cache_usage_perc,vllm:num_preemptions,vllm:prefix_cache_hits,vllm:prefix_cache_queries,vllm:time_to_first_token_seconds, andvllm:request_queue_time_secondswere confirmed. Verified 2026-08-16. - vLLM CLI Reference (docs.vllm.ai) — where
vllm bench serve,vllm bench latency,vllm bench throughput, and the flags written in this post were confirmed. Verified 2026-08-16. - vLLM Optimization and Tuning (docs.vllm.ai) — the source for how to respond to preemption and for guidance on tuning batched-token settings. Verified 2026-08-16.
- Efficient Memory Management for Large Language Model Serving with PagedAttention, arXiv:2309.06180 — the original PagedAttention paper.
- vLLM internals series — the series that works through the vLLM internals summarized here one at a time, against the official docs and the source.
LLM serving optimization sits at the intersection of hardware, algorithms, and systems software. PagedAttention borrowed from operating system design. FlashAttention rediscovered the principle of tiling from numerical linear algebra. Speculative decoding revived draft-and-verify ideas from branch prediction. The engineers who will build the next generation of LLM serving systems are those who understand not just the current tools but the first-principles reasoning behind them.