Tag: #tokenizer
Writing on GPUs, LLMs, MLOps, Kubernetes — and mindset · 4 posts
Open Models That Support Korean, and the Cost of Tokenization
A line saying a model supports Korean means it processes the language, not that it is good at it. This post lays out the card values read on 2026-08-12 for Korean-focused, bilingual, and multilingual open models, explain
2026-08-12 · 6 min read #ai#llm#huggingface#open-source-llm#korean-nlpInside the Tokenizer — Why Korean Costs More Tokens, and What That Costs
How byte-level BPE works, then downloading the actual tokenizer files of Qwen3, DeepSeek-V3, and Mixtral to tokenize the same English and Korean text and compare. Covers the vocabulary-size tradeoff, why the config vocab
2026-08-12 · 6 min read #ai-papers#model-internals#tokenizer#bpe#korean-nlpHow to Actually Read a Model Card — Pulling Out What You Need in 5 Minutes
A model card is written in a way that, read top to bottom, keeps you from finding what you need. The benchmark table takes up half the screen while the license and chat template pass by in a single line. This post flips
2026-08-02 · 19 min read #llm#huggingface#model-card#license#tokenizerBuilding an LLM from Scratch — A Stanford CS336 Style Learning Roadmap
Stanford CS336 (Language Modeling from Scratch) keeps hitting the Hacker News front page, and from-scratch LLM building is hot again. This post dissects the full curriculum — tokenizers, attention, distributed training,
2026-06-12 · 15 min read #llm#transformer#cs336#deep-learning#tokenizer