Tag: #megatron-lm
Writing on GPUs, LLMs, MLOps, Kubernetes — and mindset · 2 posts
Distributed Training & GPU Infrastructure 2026 Deep-Dive — DeepSpeed, FSDP2, Megatron-LM, Ray Train, JAX, TorchTitan, Blackwell GB200, MI325X, TPU v5p
A comparison of DeepSpeed, FSDP2, Megatron-LM, Ray Train, JAX, TorchTitan, and Composer — plus NVIDIA Blackwell GB200 NVL72, AMD MI325X, Intel Gaudi 3, AWS Trainium 2, and Google TPU v5p/v6e Trillium. 3D parallelism, ZeR
2026-05-16 · 14 min read #distributed-training#deepspeed#fsdp#megatron-lm#rayLarge-Scale Model Training Complete Guide: Strategies for Pre-training 100B+ Parameter LLMs
A complete practical guide to training hundred-billion parameter LLMs. Covers scaling laws (Chinchilla), Megatron-LM, 3D parallelism, checkpointing strategies, training stability, and data mixing with hands-on examples.
2026-03-17 · 21 min read #large-scale-training#llm#megatron-lm#distributed-training#scaling-laws