Distributed Deep Learning

Broken Symmetry in BF16 Attention: Why FlashAttention Gradients Blow Up Late in Training
BF16 is now standard in large-scale pretraining, including in fused attention kernels such as FlashAttention, and these kernels are …
Junlin Chen, Daize Dong, Huanwei Di, Haolong Jia, Jiawei Wu, Haotian Xie, Mingkai Zheng, Yang Li, Leshang Chen, Huishu Wang, Eric P. Xing, Hongyi Wang
PDF Cite arXiv
PHOENIX: Resilient LLM Training with Hot-Swapping via Zero-Overhead Checkpoint
State-of-the-art large language model (LLM) training takes tens of thousands of graphics processing units (GPUs) for months and …
Haotian Xie, Junlin Chen, Mingkai Zheng, Lishan Yang, Zhao Zhang
PDF Cite arXiv
SCAPE: Accurate and Efficient LLM Training with Extreme Sparse Communication
Communication increasingly dominates the cost of Large Language Model (LLM) pre-training, especially under data-parallel and sharded …
Mingkai Zheng, Junlin Chen, Haotian Xie, Zhao Zhang
PDF Cite arXiv
Efficient Fine-Grained GPU Performance Modeling for Distributed Deep Learning of LLM
Training Large Language Models (LLMs) is one of the most compute-intensive tasks in high-performance computing. Predicting end-to-end …
Biyao Zhang, Mingkai Zheng, Debargha Ganguly, Xuecen Zhang, Vikash Singh, Vipin Chaudhary, Zhao Zhang
PDF Cite DOI arXiv
Radius: Range-based Gradient Sparsity for Large Foundation Model Pre-training
We present Radius, a gradient sparsity algorithm and system to accelerate large foundation model (FM) training while preserving …
Mingkai Zheng, Zhao Zhang
PDF Cite Proceedings OpenReview
COMPSO: Optimizing Gradient Compression for Distributed Training with Second-Order Optimizers
Second-order optimization methods have been developed to enhance convergence and generalization in deep neural network (DNN) training …
Baixi Sun, Weijin Liu, J. Gregory Pauloski, Jiannan Tian, Jinda Jia, Daoce Wang, Boyuan Zhang, Mingkai Zheng, Sheng Di, Sian Jin, Zhao Zhang, Xiaodong Yu, Kamil A. Iskra, Pete Beckman, Guangming Tan, Dingwen Tao
Cite DOI