Tag: #rmsnorm
Writing on GPUs, LLMs, MLOps, Kubernetes — and mindset · 1 posts
Normalization and Activation — Keeping Training From Falling Apart
Confirming from config values why RMSNorm, pre-norm, and SwiGLU became the defaults, then walking through the newer devices that stop attention logits from exploding — the QK-Norm of Qwen3 and the QK-Clip of Kimi K2 — wi
2026-08-12 · 6 min read #ai-papers#model-internals#rmsnorm#swiglu#training-stability