LLM Training

Broken Symmetry in BF16 Attention: Why FlashAttention Gradients Blow Up Late in Training
BF16 is now standard in large-scale pretraining, including in fused attention kernels such as FlashAttention, and these kernels are …
Junlin Chen, Daize Dong, Huanwei Di, Haolong Jia, Jiawei Wu, Haotian Xie, Mingkai Zheng, Yang Li, Leshang Chen, Huishu Wang, Eric P. Xing, Hongyi Wang
PDF Cite arXiv
PHOENIX: Resilient LLM Training with Hot-Swapping via Zero-Overhead Checkpoint
State-of-the-art large language model (LLM) training takes tens of thousands of graphics processing units (GPUs) for months and …
Haotian Xie, Junlin Chen, Mingkai Zheng, Lishan Yang, Zhao Zhang
PDF Cite arXiv
SCAPE: Accurate and Efficient LLM Training with Extreme Sparse Communication
Communication increasingly dominates the cost of Large Language Model (LLM) pre-training, especially under data-parallel and sharded …
Mingkai Zheng, Junlin Chen, Haotian Xie, Zhao Zhang
PDF Cite arXiv
Efficient Fine-Grained GPU Performance Modeling for Distributed Deep Learning of LLM
Training Large Language Models (LLMs) is one of the most compute-intensive tasks in high-performance computing. Predicting end-to-end …
Biyao Zhang, Mingkai Zheng, Debargha Ganguly, Xuecen Zhang, Vikash Singh, Vipin Chaudhary, Zhao Zhang
PDF Cite DOI arXiv
Radius: Range-based Gradient Sparsity for Large Foundation Model Pre-training
We present Radius, a gradient sparsity algorithm and system to accelerate large foundation model (FM) training while preserving …
Mingkai Zheng, Zhao Zhang
PDF Cite Proceedings OpenReview