Tag: #distributed-training
Writing on GPUs, LLMs, MLOps, Kubernetes — and mindset · 16 posts
The Four Kinds of Multi-GPU Parallelism — What You Split and What You Communicate
A numbers-first breakdown of what data parallelism, tensor parallelism, pipeline parallelism, and context parallelism each split, and what they pay in communication for it. It starts by building the per-parameter 16-byte
2026-08-02 · 13 min read #mlops#distributed-training#multi-gpu#fsdp#deepspeedRunning a GPU Cluster with Slurm — Knowing Why a Job Will Not Run Matters More Than Submitting
Everything you need to actually use Slurm on a GPU cluster in practice. Sets up the coordinate system of partition, QoS, and account first, then covers how to request GPUs, CPUs, and memory in an sbatch script and the bi
2026-08-02 · 14 min read #mlops#slurm#hpc#gpu-cluster#distributed-trainingtorchcomms in PyTorch 2.13 — The New Distributed Communication Backend Coming for c10d
Among the highlights of PyTorch 2.13, released on July 8, 2026, the most structural change is torchcomms — PyTorch Distributed's new communication backend has begun landing in core's CI and DeviceMesh paths. torchcomms i
2026-07-17 · 10 min read #pytorch#distributed-training#nccl#deep-learningThe Workload/PodGroup API in Kubernetes v1.36 — Gang Scheduling Is Moving Into kube-scheduler
Gang scheduling for AI training and batch workloads has so far been the job of external schedulers like Volcano or Kueue, but Kubernetes has started pulling this capability into its core. v1.35 shipped the Workload API a
2026-07-16 · 19 min read #kubernetes#scheduling#gang-scheduling#distributed-training#draMulti-GPU, Multi-Node Training Platforms: The Complete Map — from the Framework Ecosystem to Hands-On Slurm and Kubeflow Guides
A one-page map of the full landscape of training models across multiple GPUs and multiple nodes. The AI library and framework ecosystem map (PyTorch, JAX, HuggingFace, DeepSpeed, Ray), when to pick which parallelization
2026-07-09 · 8 min read #ai#ml#distributed-training#slurm#kubeflowAI Interconnect — NVLink, NVSwitch, UALink, and the Art of Scaling Up
In large-scale AI training and inference, the real bottleneck is not compute but communication. This article walks through the principles and practice of AI interconnect — the scale-up domains built by NVLink and NVSwitc
2026-06-16 · 29 min read #gpu-cuda#nvlink#nvswitch#ualink#interconnectDistributed Training & GPU Infrastructure 2026 Deep-Dive — DeepSpeed, FSDP2, Megatron-LM, Ray Train, JAX, TorchTitan, Blackwell GB200, MI325X, TPU v5p
A comparison of DeepSpeed, FSDP2, Megatron-LM, Ray Train, JAX, TorchTitan, and Composer — plus NVIDIA Blackwell GB200 NVL72, AMD MI325X, Intel Gaudi 3, AWS Trainium 2, and Google TPU v5p/v6e Trillium. 3D parallelism, ZeR
2026-05-16 · 14 min read #distributed-training#deepspeed#fsdp#megatron-lm#rayTorch-Titan Complete Guide: Everything About Large-Scale Distributed Training with PyTorch
A complete guide to mastering large-scale LLM distributed training with PyTorch Titan (torchtitan). Covers FSDP2, pipeline parallelism, tensor parallelism, 4D parallelism, Flash Attention, and mixed precision with hands-
2026-03-17 · 19 min read #torch-titan#distributed-training#pytorch#fsdp#deep-learningFederated Learning Complete Guide: Privacy-Preserving Distributed AI
A complete guide to mastering Federated Learning. Learn how to train distributed models while preserving data privacy, covering FedAvg, FedProx, differential privacy, and the Flower framework with hands-on code.
2026-03-17 · 23 min read #federated-learning#privacy#distributed-training#differential-privacy#aiDeepSpeed Complete Guide: ZeRO Optimization and Large-Scale Model Training
A complete guide to mastering Microsoft DeepSpeed. Learn ZeRO-1/2/3 optimization, Offload, pipeline parallelism, mixed precision, MoE, and DeepSpeed Inference with practical configurations and code examples.
2026-03-17 · 15 min read #deepspeed#zero-optimization#distributed-training#llm#pytorchLarge-Scale Model Training Complete Guide: Strategies for Pre-training 100B+ Parameter LLMs
A complete practical guide to training hundred-billion parameter LLMs. Covers scaling laws (Chinchilla), Megatron-LM, 3D parallelism, checkpointing strategies, training stability, and data mixing with hands-on examples.
2026-03-17 · 21 min read #large-scale-training#llm#megatron-lm#distributed-training#scaling-lawsDeep Learning Training Methods Complete Guide: From Optimization to Distributed Training
A comprehensive guide covering all techniques for effectively training deep learning models. Learn gradient descent, optimizers, learning rate scheduling, regularization, batch normalization, transfer learning, fine-tuni
2026-03-17 · 28 min read #deep-learning#training#optimization#regularization#distributed-trainingRing Attention Paper Analysis: Implementing Infinite Context Window Training in Distributed Environments
Analyzes the Ring Attention paper exploring methods to overcome context length limitations in distributed environments. Covers the connection with Blockwise Parallel Transformer, implementation details, performance bench
2026-03-08 · 33 min read #ai-papers#ring-attention#distributed-training#long-context#transformerComplete Guide to Multi-GPU Distributed Training: DDP, FSDP, DeepSpeed
Systematically analyze the core components of multi-GPU distributed training including DDP, FSDP, and DeepSpeed ZeRO based on PyTorch official documentation, with practical setup instructions.
2026-03-01 · 21 min read #gpu#cuda#distributed-training#deep-learning#pytorchMastering Slurm: A Practical Guide to the HPC/AI Cluster Workload Manager
A comprehensive, hands-on guide to the Slurm workload manager. Covers architecture (slurmctld/slurmd/slurmdbd), core concepts (Partitions/QoS/Fairshare), essential commands (sbatch/srun/salloc), GPU scheduling (GRES/MIG/
2026-03-01 · 15 min read #slurm#hpc#gpu#distributed-training#clusterKubernetes AI Training Pipeline: Analyzing Volcano, Training Operator, and Kueue
A deep dive into building distributed AI training pipelines on Kubernetes based on the official documentation of Volcano, Kubeflow Training Operator, and Kueue.
2026-03-01 · 21 min read #kubernetes#ai#distributed-training#volcano#kubeflow