Tag: #model-internals

한국어English日本語中文

← All posts

• The Training Recipe — From Pre-training to Post-training, and What Reports Write Down

  Comparing, exactly as written in the reports, the three stages and six context extensions of Llama 3, the three-stage pre-training of Qwen3, the learning-rate schedule and two-phase YaRN extension of DeepSeek-V3, and the

  2026-08-12 · 6 min read #ai-papers#model-internals#pretraining#training-recipe#data-mixture

• Positional Encoding — RoPE and the Price of Context Extension

  Showing through wavelength arithmetic how a single ropetheta value acts on context length, then comparing the 500000 of Llama 3, the ABF step in Qwen3, the YaRN settings of DeepSeek-V3 and Kimi K2, and the partial rotati

  2026-08-12 · 6 min read #ai-papers#model-internals#rope#positional-encoding#long-context

• Anatomy of config.json — Reading a Model From One Settings File

  How every field in config.json — hiddensize, numhiddenlayers, numattentionheads versus numkeyvalueheads, headdim, intermediatesize, ropetheta, vocabsize, tiewordembeddings — shows up in memory and speed, and a hand count

  2026-08-12 · 6 min read #ai-papers#model-internals#config-json#transformer#llm-architecture

• Reading a Technical Report Critically — What Gets Written Down and What Goes Missing

  How to separate the verifiable from the unverifiable in a model technical report. Covers the limits of self-reported benchmarks, the places where a config and a report disagree, values that could not be read from gated r

  2026-08-12 · 7 min read #ai-papers#model-internals#tech-report#benchmarks#evaluation

• MoE Routing — How an Expert Gets Picked

  Reading the config fields of a mixture-of-experts layer against real models. Comparing Mixtral 2-of-8, Qwen3 8-of-128, the 256 routed experts plus a shared expert in DeepSeek-V3, and the sparsity of 48 in Kimi K2, then w

  2026-08-12 · 6 min read #ai-papers#model-internals#mixture-of-experts#moe#routing

• Inside the Tokenizer — Why Korean Costs More Tokens, and What That Costs

  How byte-level BPE works, then downloading the actual tokenizer files of Qwen3, DeepSeek-V3, and Mixtral to tokenize the same English and Korean text and compare. Covers the vocabulary-size tradeoff, why the config vocab

  2026-08-12 · 6 min read #ai-papers#model-internals#tokenizer#bpe#korean-nlp

• Normalization and Activation — Keeping Training From Falling Apart

  Confirming from config values why RMSNorm, pre-norm, and SwiGLU became the defaults, then walking through the newer devices that stop attention logits from exploding — the QK-Norm of Qwen3 and the QK-Clip of Kimi K2 — wi

  2026-08-12 · 6 min read #ai-papers#model-internals#rmsnorm#swiglu#training-stability

• Attention Variants — From MHA to MLA, and How the KV Cache Shrinks

  Comparing MHA, MQA, GQA, and MLA using real config values. Working out with formulas and numbers how the grouped-query attention in Qwen3, Mixtral, and GLM-4.5 and the latent attention in DeepSeek-V3 and Kimi K2 reduce p

  2026-08-12 · 6 min read #ai-papers#model-internals#attention#gqa#mla