Checkpointing

PHOENIX: Resilient LLM Training with Hot-Swapping via Zero-Overhead Checkpoint
State-of-the-art large language model (LLM) training takes tens of thousands of graphics processing units (GPUs) for months and …
Haotian Xie, Junlin Chen, Mingkai Zheng, Lishan Yang, Zhao Zhang
PDF Cite arXiv