Tag: #moe
Writing on GPUs, LLMs, MLOps, Kubernetes — and mindset · 16 posts
Text LLM Technical Reports: What to Read, and How to Read Design Decisions Instead of Rankings
Nine text-LLM technical reports, each verified by opening the arXiv abstract page directly. What DeepSeek-V3, DeepSeek-R1, Qwen3, Gemma 3, Olmo 3, Kimi K2, MiniMax-01, Mellum2 and s1 each did that was new, and which limi
2026-08-12 · 7 min read #ai-papers#paper-review#technical-report#llm#moeMoE Routing — How an Expert Gets Picked
Reading the config fields of a mixture-of-experts layer against real models. Comparing Mixtral 2-of-8, Qwen3 8-of-128, the 256 routed experts plus a shared expert in DeepSeek-V3, and the sparsity of 48 in Kimi K2, then w
2026-08-12 · 6 min read #ai-papers#model-internals#mixture-of-experts#moe#routingHow These Models Were Actually Built — Dissecting the 2026 Open-Weight Pipeline
Reading the cards and technical reports of the open-weight models sitting near the top of Hugging Face as of August 2026, this post lays out the production pipeline in order, from data collection to quantized deployment.
2026-08-02 · 17 min read #llm#pretraining#moe#post-training#quantizationHow a 26B Model Runs in 2GB of RAM — Resident Memory and Working Set Are Not the Same Number
TurboFieldfare, posted to Show HN on July 29, 2026, claims to run Gemma 4 26B-A4B on M-series Macs in about 2GB of RAM. 14.3GB is installed on disk, the only thing resident in RAM is a 1.35GB shared core, and the expert
2026-07-31 · 12 min read #ai#llm#quantization#apple-silicon#moeWhat DeepSeek V4 Flash Actually Changes — A Cost-per-Capability Question, Not a Leaderboard One
On July 31, 2026, DeepSeek moved its V4-Flash API into public beta. The architecture is identical to April's preview — 284B total parameters, 13B active MoE, 1M context — and the only thing that changed is post-training.
2026-07-31 · 11 min read #ai#llm#deepseek#inference-cost#moeTencent Hy3: reading a 295B open-weight MoE without the hype
On July 6, 2026, Tencent released Hy3 under Apache 2.0: a Mixture-of-Experts reasoning and agent model with 295B total parameters but only 21B active per token. Here is what is genuinely new, where it sits in the Chinese
2026-07-11 · 5 min read #hy3#tencent#hunyuan#open-weights#moeRunning GLM-5.2 on a slow computer — how colibrì streams a 744B model from disk
colibrì is a ~1,300-line pure-C inference engine that runs GLM-5.2, a 744B-parameter MoE model, on a consumer PC with 25GB of RAM. The trick is MoE sparsity plus disk streaming: only ~9.9GB of dense layers stay resident,
2026-07-11 · 5 min read #ai#llm#local-inference#moe#quantizationTop LLM Papers 2024-2026 - Llama, DeepSeek, Qwen, Mistral, Phi, RLHF, DPO, CoT, RAG, FlashAttention, vLLM Reading List
A curated reading list of 30+ must-read LLM papers for engineers building with LLMs in 2024-2026. Covers foundation models (Llama 3/4, DeepSeek-V3/R1, Qwen3, Mistral, Phi-4, Gemma 3), training innovations (MoE, MLA, GQA)
2026-05-16 · 19 min read #llm#papers#llama#deepseek#qwenFoundation Model Architectures 2026 — Beyond the Transformer / Mamba 2 / Hyena / RWKV / RetNet / Griffin / Jamba / xLSTM / TTT / DiT / MoE / Flash Attention 3 Deep Dive
In 2026 the foundation-model world is no longer Transformer-only. Vaswani 2017 "Attention is All You Need" remains the standard, but next to it stand state-space models (Mamba, Mamba 2), the linear-RNN renaissance (RWKV,
2026-05-16 · 22 min read #foundation-models#transformer#attention-is-all-you-need#vaswani#mamba2025 Open Source AI Models Showdown: DeepSeek R1 vs Llama 4 vs Qwen 3 vs Mistral
DeepSeek R1 (671B/37B), Llama 4 Scout/Maverick, Qwen 3 (235B MoE), Mistral 8x22B — complete comparison of the 2025 open-source AI model leaders with benchmarks, licenses, deployment guides, and cost analysis.
2026-03-22 · 20 min read #open-source#ai#llm#deepseek#llama2025 AI Research Trends: Top HuggingFace Papers and 10 Defining Research Directions
A developer-focused review of HuggingFace trending papers and the 10 defining AI research trends of 2025. DeepSeek-R1 pure RL reasoning, Nemotron-Cascade 30B/3B MoE, GRPO, PagedAttention, million-token context limitation
2026-03-21 · 15 min read #ai-research#papers#huggingface#reasoning#moeMixture of Experts (MoE) Architecture Paper Deep Analysis: From GShard to DeepSeek-MoE
Analyzes core papers on Mixture of Experts architecture, comparing routing strategies and training stability techniques across GShard, Switch Transformer, Mixtral, and DeepSeek-MoE.
2026-03-14 · 34 min read #ai-papers#moe#transformer#deepseekMixture of Experts (MoE) Architecture Deep Analysis: Evolution from Switch Transformer to Mixtral and Efficient Scaling Strategies
Deep analysis from the core principles of MoE architecture to Switch Transformer single expert routing, Mixtral 8x7B Sparse MoE, and DeepSeek-MoE fine-grained strategy. Covers routing mechanisms, load balancing loss, tra
2026-03-11 · 17 min read #ai-papers#moe#switch-transformer#mixtral#model-architectureDeep Dive into Mixture of Experts (MoE) Architecture: From Switch Transformer to Mixtral and DeepSeek
A comprehensive analysis of Mixture of Experts (MoE) architecture. Covers the mathematical foundations of Sparse MoE, routing strategies in Switch Transformer, Mixtral 8x7B, and DeepSeek-V3, training stability techniques
2026-03-10 · 15 min read #ai-papers#moe#transformer#mixtral#deepseekDeep Dive into Sparse Mixture of Experts (MoE) Architecture: From Design Principles to DeepSeek-V3 and Qwen3
Analyzing the mathematical principles, routing strategies, and load balancing techniques of Sparse MoE architecture, covering the design choices and practical training/inference optimization of modern MoE models from Swi
2026-03-06 · 22 min read #ai-papers#moe#sparse-model#deepseek#2026-03Mixture of Experts (MoE) Architecture: A Complete Analysis
A complete analysis of MoE architectures, from the principles of Sparse MoE to the MoE implementations in Mixtral and DeepSeek-V3, routing strategies, and load balancing.
2026-03-03 · 6 min read #ai-papers#moe#mixtral#deepseek#2026-03