Tag: #bpe
Writing on GPUs, LLMs, MLOps, Kubernetes — and mindset · 1 posts
Inside the Tokenizer — Why Korean Costs More Tokens, and What That Costs
How byte-level BPE works, then downloading the actual tokenizer files of Qwen3, DeepSeek-V3, and Mixtral to tokenize the same English and Korean text and compare. Covers the vocabulary-size tradeoff, why the config vocab
2026-08-12 · 6 min read #ai-papers#model-internals#tokenizer#bpe#korean-nlp