Tag: #preemption
Writing on GPUs, LLMs, MLOps, Kubernetes — and mindset · 1 posts
Inside vLLM (4) — The Scheduler and Preemption, Where Throughput Collapses
Explains what the vLLM scheduler decides on every step, how the waiting queue and running list move, and how preemption, triggered when the KV cache runs short, wrecks latency and throughput, verified against the officia
2026-08-12 · 7 min read #vllm#scheduler#preemption#kv-cache#llm