Tag: #multi-gpu
Writing on GPUs, LLMs, MLOps, Kubernetes — and mindset · 1 posts
The Four Kinds of Multi-GPU Parallelism — What You Split and What You Communicate
A numbers-first breakdown of what data parallelism, tensor parallelism, pipeline parallelism, and context parallelism each split, and what they pay in communication for it. It starts by building the per-parameter 16-byte
2026-08-02 · 13 min read #mlops#distributed-training#multi-gpu#fsdp#deepspeed