Tag: #alibi
Writing on GPUs, LLMs, MLOps, Kubernetes — and mindset · 1 posts
Understanding Positional Encoding — From Sine Waves to RoPE
Starting from why Transformers need positional information, this article explains sinusoidal, learned, and relative positional encodings step by step, then RoPE and ALiBi. It connects length extrapolation and context ext
2026-06-26 · 17 min read #llm#positional-encoding#rope#alibi#long-context