Reduce-scatter
Workers circulate chunks and add partial gradients. After N-1 steps, each worker owns one fully reduced chunk.
Watch workers exchange gradient chunks around a ring, then change worker count, payload size, bandwidth, and link latency to see when distributed training scales.
Workers circulate chunks and add partial gradients. After N-1 steps, each worker owns one fully reduced chunk.
The reduced chunks circulate for another N-1 steps until every worker reconstructs the complete gradient.
Each phase needs N-1 synchronized rounds. Small payloads or many workers can become latency-bound.