Gradient Sparsity

Radius: Range-based Gradient Sparsity for Large Foundation Model Pre-training

We present Radius, a gradient sparsity algorithm and system to accelerate large foundation model (FM) training while preserving …

Mingkai Zheng, Zhao Zhang