Tag: #mixtral
Writing on GPUs, LLMs, MLOps, Kubernetes — and mindset · 4 posts
Foundation Model Architectures 2026 — Beyond the Transformer / Mamba 2 / Hyena / RWKV / RetNet / Griffin / Jamba / xLSTM / TTT / DiT / MoE / Flash Attention 3 Deep Dive
In 2026 the foundation-model world is no longer Transformer-only. Vaswani 2017 "Attention is All You Need" remains the standard, but next to it stand state-space models (Mamba, Mamba 2), the linear-RNN renaissance (RWKV,
2026-05-16 · 22 min read #foundation-models#transformer#attention-is-all-you-need#vaswani#mambaMixture of Experts (MoE) Architecture Deep Analysis: Evolution from Switch Transformer to Mixtral and Efficient Scaling Strategies
Deep analysis from the core principles of MoE architecture to Switch Transformer single expert routing, Mixtral 8x7B Sparse MoE, and DeepSeek-MoE fine-grained strategy. Covers routing mechanisms, load balancing loss, tra
2026-03-11 · 17 min read #ai-papers#moe#switch-transformer#mixtral#model-architectureDeep Dive into Mixture of Experts (MoE) Architecture: From Switch Transformer to Mixtral and DeepSeek
A comprehensive analysis of Mixture of Experts (MoE) architecture. Covers the mathematical foundations of Sparse MoE, routing strategies in Switch Transformer, Mixtral 8x7B, and DeepSeek-V3, training stability techniques
2026-03-10 · 15 min read #ai-papers#moe#transformer#mixtral#deepseekMixture of Experts (MoE) Architecture: A Complete Analysis
A complete analysis of MoE architectures, from the principles of Sparse MoE to the MoE implementations in Mixtral and DeepSeek-V3, routing strategies, and load balancing.
2026-03-03 · 6 min read #ai-papers#moe#mixtral#deepseek#2026-03