Tag: #gqa
Writing on GPUs, LLMs, MLOps, Kubernetes — and mindset · 3 posts
Attention Variants — From MHA to MLA, and How the KV Cache Shrinks
Comparing MHA, MQA, GQA, and MLA using real config values. Working out with formulas and numbers how the grouped-query attention in Qwen3, Mixtral, and GLM-4.5 and the latent attention in DeepSeek-V3 and Kimi K2 reduce p
2026-08-12 · 6 min read #ai-papers#model-internals#attention#gqa#mlaThe Evolution of Attention — MQA, GQA, FlashAttention, and Long Context
We analyze the memory and compute cost of standard attention, then explain how MQA and GQA shrink the KV cache and how FlashAttention optimizes IO. We compare sliding-window and long-context techniques and trace how all
2026-06-26 · 18 min read #llm#attention#flashattention#gqa#mqaKV Cache Optimization Deep Dive: GQA, MLA, and MHA Attention Mechanisms with Memory Efficiency Strategies
From Transformer Self-Attention KV Cache fundamentals through MHA, MQA, GQA (Llama 2/3), and MLA (DeepSeek-V2/V3) mechanism memory analysis and comparison, KV Cache compression techniques (quantization, eviction policies
2026-03-11 · 17 min read #ai-papers#kv-cache#attention-mechanism#gqa#mla