Tag: #llm-training
Writing on GPUs, LLMs, MLOps, Kubernetes — and mindset · 3 posts
Lessons from Published Training Runs — What Was Tried, and What Failed
Seven published large-scale training technical reports and logbooks, stripped down to just the failures and the responses to them, not the scoreboard. The statistics of 419 interruptions over 54 days from Llama 3 405B le
2026-08-02 · 13 min read #mlops#llm-training#case-study#training-stability#scalingA Map of the LLM Training Stack in 2026 — What Each Layer Does For You, and What It Hides
Sorts LLM training frameworks into three layers and maps out their lineage. The bottom layer is the execution engines — PyTorch distributed, DeepSpeed, Megatron-Core. The middle layer is training loops like torchtitan an
2026-08-02 · 13 min read #mlops#llm-training#pytorch#trl#frameworkDistributed Training & GPU Infrastructure 2026 Deep-Dive — DeepSpeed, FSDP2, Megatron-LM, Ray Train, JAX, TorchTitan, Blackwell GB200, MI325X, TPU v5p
A comparison of DeepSpeed, FSDP2, Megatron-LM, Ray Train, JAX, TorchTitan, and Composer — plus NVIDIA Blackwell GB200 NVL72, AMD MI325X, Intel Gaudi 3, AWS Trainium 2, and Google TPU v5p/v6e Trillium. 3D parallelism, ZeR
2026-05-16 · 14 min read #distributed-training#deepspeed#fsdp#megatron-lm#ray