LLM Training | Mingkai Zheng's Homepage

LLM Training | Mingkai Zheng's Homepagehttps://zhengmk.github.io/tag/llm-training/LLM TrainingWowchemy (https://wowchemy.com)en-usMon, 28 Sep 2026 00:00:00 +0000https://zhengmk.github.io/media/icon_hu0158c3d4eaa2a6b730ca31a67c18838d_142122_512x512_fill_lanczos_center_2.pngLLM Traininghttps://zhengmk.github.io/tag/llm-training/Broken Symmetry in BF16 Attention: Why FlashAttention Gradients Blow Up Late in Traininghttps://zhengmk.github.io/publication/broken-symmetry-in-bf16-attention-why-flashattention-gradients-blow-up-late-in-training/Mon, 28 Sep 2026 00:00:00 +0000https://zhengmk.github.io/publication/broken-symmetry-in-bf16-attention-why-flashattention-gradients-blow-up-late-in-training/PHOENIX: Resilient LLM Training with Hot-Swapping via Zero-Overhead Checkpointhttps://zhengmk.github.io/publication/phoenix-resilient-llm-training-with-hot-swapping-via-zero-overhead-checkpoint/Thu, 02 Jul 2026 00:00:00 +0000https://zhengmk.github.io/publication/phoenix-resilient-llm-training-with-hot-swapping-via-zero-overhead-checkpoint/SCAPE: Accurate and Efficient LLM Training with Extreme Sparse Communicationhttps://zhengmk.github.io/publication/scape-accurate-and-efficient-llm-training-with-extreme-sparse-communication/Thu, 02 Jul 2026 00:00:00 +0000https://zhengmk.github.io/publication/scape-accurate-and-efficient-llm-training-with-extreme-sparse-communication/Efficient Fine-Grained GPU Performance Modeling for Distributed Deep Learning of LLMhttps://zhengmk.github.io/publication/efficient-fine-grained-gpu-performance-modeling-for-distributed-deep-learning-of-llm/Wed, 17 Dec 2025 00:00:00 +0000https://zhengmk.github.io/publication/efficient-fine-grained-gpu-performance-modeling-for-distributed-deep-learning-of-llm/Radius: Range-based Gradient Sparsity for Large Foundation Model Pre-traininghttps://zhengmk.github.io/publication/radius-range-based-gradient-sparsity-for-large-foundation-model-pre-training/Mon, 12 May 2025 00:00:00 +0000https://zhengmk.github.io/publication/radius-range-based-gradient-sparsity-for-large-foundation-model-pre-training/