Make the Parallel Truly Parallel: Interleaving Parallel ZeRO-DP for RLHF Training with ZeroFuse

Wenhao Huang, Yuhao Zhang, Laiping Zhao, Yiming Li, Yichi Chen, Yanjie Wang, Fupeng Li, Yihan Wang, Zhihang Tang, Deze Zeng, Keqiu Li

October 2026

Type

Journal article

Publication

IEEE Transactions on Computers ( Early Access )