Integrating RCCL GPU-Initiated Networking into TorchTitan’s MoE Communication

Описание

Hi, I'm Rishi from AMD. Within distributed training, communication is one of the primary bottlenecks, and limiting the time spent on communication can massively boost end-to-end performance. We investigate this hypothesis by integrating work from the Rickle team to use…