Tag: #ring-attention
Writing on GPUs, LLMs, MLOps, Kubernetes — and mindset · 3 posts
Foundation Model Architectures 2026 — Beyond the Transformer / Mamba 2 / Hyena / RWKV / RetNet / Griffin / Jamba / xLSTM / TTT / DiT / MoE / Flash Attention 3 Deep Dive
In 2026 the foundation-model world is no longer Transformer-only. Vaswani 2017 "Attention is All You Need" remains the standard, but next to it stand state-space models (Mamba, Mamba 2), the linear-RNN renaissance (RWKV,
2026-05-16 · 22 min read #foundation-models#transformer#attention-is-all-you-need#vaswani#mambaRing Attention Paper Analysis: Implementing Infinite Context Window Training in Distributed Environments
Analyzes the Ring Attention paper exploring methods to overcome context length limitations in distributed environments. Covers the connection with Blockwise Parallel Transformer, implementation details, performance bench
2026-03-08 · 33 min read #ai-papers#ring-attention#distributed-training#long-context#transformerComplete Guide to LLM Long-Context Performance and KV Cache Optimization: From MQA to Ring Attention
A comprehensive practical guide covering KV Cache fundamentals for LLM long-context processing, memory consumption analysis, optimization techniques including MQA, GQA, PagedAttention, sliding window attention, and Ring
2026-03-07 · 17 min read #llm#kv-cache#long-context#multi-query-attention#grouped-query-attention