Tag: #mla
Writing on GPUs, LLMs, MLOps, Kubernetes — and mindset · 2 posts
Attention Variants — From MHA to MLA, and How the KV Cache Shrinks
Comparing MHA, MQA, GQA, and MLA using real config values. Working out with formulas and numbers how the grouped-query attention in Qwen3, Mixtral, and GLM-4.5 and the latent attention in DeepSeek-V3 and Kimi K2 reduce p
2026-08-12 · 6 min read #ai-papers#model-internals#attention#gqa#mlaKV Cache Optimization Deep Dive: GQA, MLA, and MHA Attention Mechanisms with Memory Efficiency Strategies
From Transformer Self-Attention KV Cache fundamentals through MHA, MQA, GQA (Llama 2/3), and MLA (DeepSeek-V2/V3) mechanism memory analysis and comparison, KV Cache compression techniques (quantization, eviction policies
2026-03-11 · 17 min read #ai-papers#kv-cache#attention-mechanism#gqa#mla