Yuxiong He

ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep learning

Olatunji Ruwase, Samyam Rajbhandari, Jeff Rasley, 2021, SC21: International Conference for High Performance Computing, Networking, Storage and Analysis.

1-bit Adam: Communication Efficient Large-Scale Training with Adam's Convergence Speed

Xiangru Lian, Ammar Ahmad Awan, Hanlin Tang, 2021, ICML.

1-bit LAMB: Communication Efficient Large-Scale Large-Batch Training with LAMB’s Convergence Speed pdf

Ammar Ahmad Awan, Hanlin Tang, Samyam Rajbhandari, 2021, 2022 IEEE 29th International Conference on High Performance Computing, Data, and Analytics (HiPC).

Curriculum Learning: A Regularization Method for Efficient and Stable Billion-Scale GPT Model Pre-Training pdf

Conglong Li, Yuxiong He, Minjia Zhang, 2021, ArXiv.

Scalable and Efficient MoE Training for Multitask Multilingual Models pdf

Alexandre Muzio, Ammar Ahmad Awan, Hany Hassan Awadalla, 2021, ArXiv.

NxMTransformer: Semi-Structured Sparsification for Natural Language Understanding via ADMM pdf

Bo Wu, Yuxiong He, Minjia Zhang, 2021, NeurIPS.

COCA: Online Resource Management for Cost Minimization and Carbon Neutrality in Data Centers

Shaolei Ren, Yuxiong He, 2013 .

ZeRO: Memory optimizations Toward Training Trillion Parameter Models

Olatunji Ruwase, Samyam Rajbhandari, Jeff Rasley, 2020, SC20: International Conference for High Performance Computing, Networking, Storage and Analysis.