Distributed Optimizer

SCAPE: Accurate and Efficient LLM Training with Extreme Sparse Communication
Communication increasingly dominates the cost of Large Language Model (LLM) pre-training, especially under data-parallel and sharded …
Mingkai Zheng, Junlin Chen, Haotian Xie, Zhao Zhang
PDF Cite arXiv