Tag: #throughput
Writing on GPUs, LLMs, MLOps, Kubernetes — and mindset · 2 posts
Making Inference Fast — Speculative Decoding and Throughput Optimization
From the fundamental reason LLM decode is slow, to how speculative decoding boosts speed, variants such as Medusa and EAGLE, chunked prefill and prefill/decode disaggregation, the latency versus throughput trade-off, and
2026-06-26 · 13 min read #speculative-decoding#throughput#inference#mlops#latencyAccelerating LLM Inference 2-3x with Speculative Decoding: From Theory to Production
A deep dive into the mathematical foundations of Speculative Decoding, the Draft-Verify pipeline, acceptance probability analysis, practical implementation in vLLM/TensorRT-LLM, and Apple's Mirror Speculative Decoding.
2026-03-02 · 8 min read #llm#speculative-decoding#inference#optimization#vllm