Sparse Communication

SCAPE: Accurate and Efficient LLM Training with Extreme Sparse Communication
Communication increasingly dominates the cost of Large Language Model (LLM) pre-training, especially under data-parallel and sharded …
Mingkai Zheng, Junlin Chen, Haotian Xie, Zhao Zhang
PDF Cite arXiv
Radius: Range-based Gradient Sparsity for Large Foundation Model Pre-training
We present Radius, a gradient sparsity algorithm and system to accelerate large foundation model (FM) training while preserving …
Mingkai Zheng, Zhao Zhang
PDF Cite Proceedings OpenReview