Tag: #korean-nlp
Writing on GPUs, LLMs, MLOps, Kubernetes — and mindset · 4 posts
Open Models That Support Korean, and the Cost of Tokenization
A line saying a model supports Korean means it processes the language, not that it is good at it. This post lays out the card values read on 2026-08-12 for Korean-focused, bilingual, and multilingual open models, explain
2026-08-12 · 6 min read #ai#llm#huggingface#open-source-llm#korean-nlpInside the Tokenizer — Why Korean Costs More Tokens, and What That Costs
How byte-level BPE works, then downloading the actual tokenizer files of Qwen3, DeepSeek-V3, and Mixtral to tokenize the same English and Korean text and compare. Covers the vocabulary-size tradeoff, why the config vocab
2026-08-12 · 6 min read #ai-papers#model-internals#tokenizer#bpe#korean-nlpComplete Guide to Korean LLM Training Data: Hugging Face Datasets, Preprocessing, and Quality Control
Everything about LLM training data! Hugging Face datasets (types/loading/conversion), Korean data collection (crawling/synthetic/translation), preprocessing (tokenization/cleaning/dedup), Instruction Tuning formats (Alpa
2026-03-25 · 23 min read #llm#training-data#huggingface#dataset#korean-nlpComplete Guide to Korean NLP and LLM: Mastering KoBERT, KLUE, HyperCLOVA, and EXAONE
A comprehensive guide to Korean NLP and LLM covering KoBERT, KLUE, HyperCLOVA, and EXAONE.
2026-03-17 · 27 min read #korean-nlp#kobert#klue#hyperclova#exaone