LLM Caching, Explained — Why Prompt Caching and Prefix Caches Save You Money

Introduction — The Mystery of "Same Prefix, Cheaper Request"
Step 1 — Inference Has Two Phases: Prefill and Decode
Step 2 — Attention and K/V: Every Token Prints a Business Card
Step 3 — The KV Cache: So Store Them
Step 4 — Prefix Cache: Reuse Across Requests
Step 5 — Provider Prompt Caching: How to Read the Price Sheet
Step 6 — Practical Design Rules
Closing — The Price Sheet Is Physics
References

Introduction — The Mystery of "Same Prefix, Cheaper Request"

Prompt caching looks strange at first. You don't get a discount for asking the same question twice — you get one when the front of your prompt matches, even if everything after it is completely different. Stranger still, some providers charge extra to write the cache and a tenth of the price to read it. That odd price sheet isn't marketing. It is a direct reflection of the physics inside a transformer.

This article climbs up from that physics: what actually happens inside a transformer during generation, what the KV cache is and why it only works on prefixes, how serving engines like vLLM and SGLang implement prefix caching, and how to design for Anthropic's and OpenAI's prompt caching in practice.

Step 1 — Inference Has Two Phases: Prefill and Decode

Transformer inference has two phases with completely different characters.

Prefill — processing the entire input prompt at once. Every prompt token can be computed in parallel, saturating the GPU's arithmetic units (compute-bound). Time-to-first-token (TTFT) is mostly decided here.
Decode — generating the answer one token at a time. Each token requires a full pass through the model, and the bottleneck is memory bandwidth, not arithmetic (memory-bound).

What prompt caching saves is precisely the prefill computation. If your 100K-token system prompt is in cache, those 100K tokens never get prefilled again. That is why a cache hit doesn't just cut cost — it dramatically cuts TTFT.

Step 2 — Attention and K/V: Every Token Prints a Business Card

Why can prefill be skipped? Look inside attention.

In every layer of a transformer, each token is transformed into three vectors — Query, Key, and Value. Intuitively: each token prints a business card (K) and a payload (V), and every new token uses its question (Q) to scan the previous tokens' business cards, compute relevance, and blend in their payloads proportionally.

One property matters above all. Attention in GPT-style models is causal — each token can only see tokens before it. Which means:

Token i's K and V are a function of tokens 1..i only. Nothing that comes later can ever change them.

That property is the root of all caching. If "the text so far" is identical, every token's K/V vectors compute to bit-identical values. So why compute them twice? Store them.

Step 3 — The KV Cache: So Store Them

The KV cache is exactly that store. During decode, the model needs the K/V of every previous token. Without a cache, each generated token would require recomputing the whole sequence so far, and generation cost would explode quadratically. With the cache, only the new token's Q/K/V are computed; everything else is read.

It isn't free — the KV cache eats memory. Per-token size is roughly:

KV cache per token = 2 (K and V) × layers × KV heads × head dim × bytes

Example: a 7B-class model (32 layers, hidden 4096, fp16)
   = 2 × 32 × 4096 × 2 bytes ≈ 512 KB per token
   → one 4,000-token conversation ≈ 2 GB
   → a few dozen concurrent users exhausts GPU memory fast

This memory pressure explains much of modern architecture: GQA (many Q heads sharing fewer K/V heads) and MQA exist to shrink the cache by an integer factor, and KV-cache quantization (FP8/INT4) comes from the same squeeze. In the long-context era, the real cost is often the KV cache, not the weights.

Step 4 — Prefix Cache: Reuse Across Requests

So far the KV cache lived within one request. A prefix cache extends it across requests.

If two requests share the same front portion (same system prompt + same document), that portion's K/V is exactly identical — thanks to causality. So instead of discarding the K/V blocks computed for the first request, keep them, and when the next request's prefix matches, continue from them with no prefill.

Why prefixes only? The flip side of causality. Since token i's K/V depends on 1..i, changing even one token in the middle changes the K/V of every token after it. Insert one timestamp into the middle of a prompt, and the cache for the 100K tokens behind it is void. The cache responds not to "contains the same content" but to "byte-identical prefix."

Real serving-engine implementations:

vLLM — Automatic Prefix Caching (APC): vLLM's PagedAttention manages the KV cache in fixed-size blocks, like OS paging. APC identifies each block by a hash of "all tokens up to this block," so an existing block with the same hash is simply reused. Being block-granular, partial overlaps work naturally.
SGLang — RadixAttention: manages shared prefixes in a radix tree. Hundreds of requests branching off one system prompt form a tree — and the data structure mirrors it exactly, which makes it especially efficient for multi-turn chat and parallel branching. Old branches are evicted LRU-style.

Step 5 — Provider Prompt Caching: How to Read the Price Sheet

Provider "prompt caching" is this prefix cache productized. Now the price sheet reads as physics.

Anthropic (Claude) — explicit caching. You mark cache_control breakpoints in the request (up to 4).

Cache write: 1.25× base input price (5-minute TTL) or 2× (1-hour TTL) — the cost of computing and retaining the K/V.
Cache read: ~0.1× base input price — prefill is skipped, hence one tenth.
Break-even: with the 5-minute TTL the second request already wins (1.25 + 0.1 < 2); the 1-hour TTL needs three or more uses (2 + 0.2 < 3).
Minimum cacheable prefix: 1024–4096 tokens depending on the model. Shorter prefixes silently don't cache even with a marker.
Render order is tools → system → messages — tools sit at the very front, so adding, removing, or reordering a tool invalidates the entire cache.

OpenAI — automatic caching. Prompts of 1024+ tokens with matching prefixes are cached without any marker, and cached input tokens are discounted 50% per the public docs. No markers means convenience — and no control over what got cached.

Verification is the same either way — read the usage fields on the response. On Anthropic, cache_read_input_tokens must be nonzero for the cache to actually be read. If it stays zero across repeated requests, a silent cache-buster is hiding somewhere.

Step 6 — Practical Design Rules

Once you know the mechanism, the rules write themselves.

Stable content first, volatile content last. Fixed system prompt, documents, and tool definitions go up front; user questions, timestamps, and request IDs go at the end. The ordering in your prompt-assembly code determines your cache performance.
Hunt the silent cache-busters. A datetime.now() embedded in the system prompt, a fresh UUID per request, unsorted JSON serialization (key order differs every time), per-user tool lists — all of them change prefix bytes and shatter the cache.
A conversation is a growing prefix. In multi-turn chat the entire previous conversation is the prefix, so cache savings grow with every turn. The standard pattern puts a breakpoint at the end of the most recently appended turn.
Switching models discards the cache. K/V values are computed by a specific model's weights, so caches are per-model. Change models mid-session and you rewrite from scratch.
Self-hosting? Turn it on. Just enabling vLLM's APC or using SGLang cuts prefill costs substantially for chat-like workloads — dramatically so for "many questions against the same document" patterns.

Closing — The Price Sheet Is Physics

Compressed to one paragraph: attention is causal, so earlier tokens' K/V are immutable; the KV cache reuses them within a request; the prefix cache reuses them across requests; and prompt caching's price sheet — write premium, read discount, prefix-only, minimum length — is that reuse's cost structure translated into prices. The answer to "why only the front part?" was never policy. It was one line of math: token i's K/V is a function of 1..i alone.

The broader serving-cost picture continues in the serving section of the AI model development lifecycle and the GPU-partitioning MIG guide.