Tag: #linear-attention
Writing on GPUs, LLMs, MLOps, Kubernetes — and mindset · 5 posts
Foundation Model Architectures 2026 — Beyond the Transformer / Mamba 2 / Hyena / RWKV / RetNet / Griffin / Jamba / xLSTM / TTT / DiT / MoE / Flash Attention 3 Deep Dive
In 2026 the foundation-model world is no longer Transformer-only. Vaswani 2017 "Attention is All You Need" remains the standard, but next to it stand state-space models (Mamba, Mamba 2), the linear-RNN renaissance (RWKV,
2026-05-16 · 22 min read #foundation-models#transformer#attention-is-all-you-need#vaswani#mambaRWKV Architecture Deep Dive: Linear Attention RNN That Rivals Transformers
A comprehensive analysis of RWKV architecture covering the WKV attention mechanism, linear complexity advantages, comparison with Transformers and Mamba, training methodology, inference optimization, and practical deploy
2026-03-07 · 19 min read #ai-papers#rwkv#linear-attention#rnn#transformerRWKV: Reinventing RNNs for the Transformer Era — From v4 to v7 Goose
Analyzing the RWKV architecture that overcomes the O(N²) limitation of Transformers. We explore the fusion of Linear Attention and RNNs, selective state space mechanisms, and the innovations in v7 Goose — all with code e
2026-03-03 · 9 min read #ai-papers#rwkv#rnn#linear-attention#state-space-modelRWKV-7 "Goose" Architecture Analysis — A Linear-Time Model Surpassing Transformers
A paper-based analysis of RWKV-7 Goose Dynamic State Evolution mechanism, TC0 barrier breakthrough, and performance comparison against Transformers. A next-generation architecture enabling constant memory + linear time i
2026-03-03 · 9 min read #ai-papers#rwkv#linear-attention#sequence-modeling#2026-03Mamba Paper Review: Going Beyond Transformers with Selective State Space Models
An in-depth review of the Mamba (Selective State Space Models) paper. Analyzing the evolution from S4 to Mamba, the Selective Scan mechanism, Hardware-Aware algorithms, and Mamba-2 State Space Duality with code examples.
2026-03-02 · 8 min read #mamba#state-space-model#ssm#transformer#linear-attention