Tag: #mqa
Writing on GPUs, LLMs, MLOps, Kubernetes — and mindset · 1 posts
The Evolution of Attention — MQA, GQA, FlashAttention, and Long Context
We analyze the memory and compute cost of standard attention, then explain how MQA and GQA shrink the KV cache and how FlashAttention optimizes IO. We compare sliding-window and long-context techniques and trace how all
2026-06-26 · 18 min read #llm#attention#flashattention#gqa#mqa