Tag: #attention

한국어English日本語中文

← All posts

• The Hybrid Attention in Qwen3.8-27B — Only 16 of 64 Layers Grow a KV Cache

  The reason a 27B model can carry a 262K-token context on a laptop is not its parameter count but its layer layout. Qwen3.8-27B places 48 of its 64 layers as linear attention (Gated DeltaNet) and only 16 as ordinary atten

  2026-08-14 · 8 min read #llm#inference#quantization#local-llm#attention

• Attention Variants — From MHA to MLA, and How the KV Cache Shrinks

  Comparing MHA, MQA, GQA, and MLA using real config values. Working out with formulas and numbers how the grouped-query attention in Qwen3, Mixtral, and GLM-4.5 and the latent attention in DeepSeek-V3 and Kimi K2 reduce p

  2026-08-12 · 6 min read #ai-papers#model-internals#attention#gqa#mla

• Optimizing Long-Context Inference with Hybrid SWA — What Xiaomi's MiMo v2.5 Actually Does

  Hybrid SWA interleaves sliding-window attention with full attention across layers to cut both KV-cache memory and compute for long-context inference — now a mainstream design. Grounded in Xiaomi's published MiMo v2.5 inf

  2026-07-11 · 7 min read #ai#llm#inference#attention#optimization

• Reading Deeply, Thinking Slowly — The Craft of Reflection

  In an age of endless skimming, we are slowly losing the habit of reading deeply. This essay looks calmly at the difference between shallow and deep reading, what the internet did to our attention, the arts of active and

  2026-07-05 · 17 min read #learning#reading#reflection#attention

• The Truth About Dopamine Detox — Focus Myths and the Science

  Dopamine is not a pleasure chemical but a signal of motivation and prediction. This piece examines the scientific basis of the trendy dopamine detox and, evidence in hand, cheerfully explores what really helps focus in a

  2026-07-01 · 28 min read #culture#dopamine#attention#focus#neuroscience

• Dissecting the Transformer — From Attention to KV Cache

  A from-scratch breakdown of the Transformer: self-attention, multi-head, positional encoding, the FFN, and residual connections with normalization. It connects tensor shapes and parameter counts, causal masking, encoder/

  2026-06-26 · 16 min read #llm#transformer#attention#positional-encoding#kv-cache

• The Evolution of Attention — MQA, GQA, FlashAttention, and Long Context

  We analyze the memory and compute cost of standard attention, then explain how MQA and GQA shrink the KV cache and how FlashAttention optimizes IO. We compare sliding-window and long-context techniques and trace how all

  2026-06-26 · 18 min read #llm#attention#flashattention#gqa#mqa

• Transformer Architecture Complete Analysis: From Attention to Modern LLMs

  A complete guide to understanding Transformer architecture from scratch. Covers Self-Attention, Multi-Head Attention, Positional Encoding, and Encoder-Decoder structure through Flash Attention, RoPE, and GQA with formula

  2026-03-17 · 22 min read #transformer#attention#deep-learning#nlp#ai

• Attention Is All You Need - A Complete Analysis of the Transformer Paper

  A thorough analysis of the "Attention Is All You Need" paper, the foundation of the Transformer architecture, breaking down its core mechanisms including Self-Attention, Multi-Head Attention, and Positional Encoding one

  2026-03-01 · 18 min read #ai-papers#transformer#deep-learning#attention