Distributed Training
Parallelism paradigms (DDP, TP, PP, FSDP) and scaling laws.
For a small example, two workers see different mini-batches. Compute local gradients, all-reduce their average, and compare the update with one combined batch.
Distributed Training
In practice
Parameters no longer fit on one accelerator. Choose data, tensor, or pipeline parallelism, estimate communication, profile utilization, and validate numerical parity.
Initializing Interactive Playground...
Experiment notes
Record your experiment
Predict how one control changes the Distributed Training result.
Implementation
Repair a failing Distributed Training starter
The starter deliberately follows a tempting but flawed assumption: Adding GPUs before profiling one worker
- Run the starter once and inspect the first wrong value, failed check, or incomplete branch.
- Repair the smallest relevant part of the code. Do not rewrite the whole solution.
- Check the solution, then explain why this correction works: Establish a stable single-device throughput and memory baseline.
Check: The browser checks pass and you can name the failure mechanism before opening the explanation.
Failing starter lab
Average gradients across ranks
Simulate an all-reduce mean for identical-shaped gradients from multiple workers.
Run the incomplete starter before opening hints. Use the first wrong output or failed check to form a repair hypothesis.
Knowledge check
10 questions
Explanations after each answer · 80% to pass · Distributed Training
Chapter 1 of 5 · Core
Data Parallelism & Ring-AllReduce Communication Topology
In distributed training, Distributed Data Parallelism (DDP) replicates the model across GPU nodes. Each GPU receives a distinct slice of the global batch (a micro-batch) and performs an independent forward and backward pass. The primary engineering challenge is the rapid aggregation of local gradients: Rather than relying on a centralized Parameter Server which suffers from bandwidth saturation ( communication overhead), modern clusters utilize Ring-AllReduce.
In a Ring-AllReduce topology, the nodes are arranged in a logical ring. Let represent the total number of parameter gradients (in bytes). The gradient array is partitioned into equal chunks. The algorithm executes in two successive phases:
- Scatter-Reduce: Over steps, each node sends a chunk to its right neighbor while receiving a chunk from its left neighbor, aggregating the incoming gradients. At the end of this phase, each node holds the fully aggregated gradient sum for exactly one chunk of size .
- All-Gather: Over another steps, the fully aggregated chunks are circulated around the ring so that every node obtains the complete set of aggregated gradients.
Communication Complexity Analysis: The total volume of data transmitted by each GPU throughout the entire Ring-AllReduce process is: Remarkably, the total data sent per GPU is independent of the number of nodes in the limit as , approaching a bound of . This ensures high linear scaling efficiency.
Further reading
Two free references selected for Distributed Training.
References
Primary sources and free technical reading for Distributed Training.