NVIDIA has unveiled the Reshard collective, a novel communication method designed to optimize weight refit in reinforcement learning (RL) workflows. Weight refit involves transferring updated policy model parameters from training GPUs to inference GPUs, a process critical for generating fresh training trajectories. The Reshard collective aims to address inefficiencies in current methods, particularly as model sizes continue to scale.

Challenges in Weight Refit for Large Models

Traditional weight refit methods rely on a gather-and-broadcast approach, where training GPUs reconstruct complete model parameters before broadcasting them to inference GPUs. While straightforward, this process results in redundant communication, as inference GPUs receive entire tensors even when only specific shards are required. For large models such as Kimi K3 (2.8 trillion parameters) and Qwen 3.8 2.4T A95B (2.4 trillion parameters), this inefficiency is magnified. Assuming BF16 weights, these models require approximately 11.2 TB and 9.6 TB of communication payload per refit, respectively.

Reshard Collective: A DTensor-to-DTensor Solution

The Reshard collective reformulates weight refit as a DTensor-to-DTensor transfer problem, leveraging PyTorch's distributed tensor abstraction. This approach enables direct shard-to-shard transfers between training and inference GPUs, eliminating the need for intermediate full-tensor representations. By mapping overlapping shard regions and optimizing communication paths, the Reshard collective minimizes payload size and balances data distribution across GPUs.

Performance Benchmarks

According to NVIDIA's internal benchmarks, the Reshard collective significantly reduces refit latency. For the 671-billion-parameter DeepSeek-V3 model, latency dropped from 14.9 seconds to 2.6 seconds on DGX H100 systems and from 6.1 seconds to 2.3 seconds on DGX GB200 systems. Additional tests on a 256-GPU NVL72 cluster with NDR InfiniBand showed a reduction in weight synchronization time from 5.78 seconds to 2.77 seconds, cutting total RL step time by 12.7%. A single feed-forward mixture-of-experts layer transfer improved from 77.3 milliseconds to 9.8 milliseconds compared to flat direct sends.

Open-Source Implementation

The Reshard collective is available as an open-source implementation through NVIDIA's NCCL extensions repository. The package includes experimental mesh-to-mesh components and dependencies on CUDA 12 and CUDA 13, with NCCL 2.30.7 validated for use. NVIDIA's documentation highlights the potential for integration into frameworks like NeMo RL and vLLM, which have previously explored weight-transfer optimizations.

By addressing the growing bottleneck of weight synchronization in RL workflows, the Reshard collective represents a critical advancement for scaling large AI models efficiently.