S. Shi

发表

Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutes pdf

Yuanzhou Yang, Shaohuai Shi, Yangzihao Wang, 2018, ArXiv.

Benchmarking State-of-the-Art Deep Learning Software Tools

Qiang Wang, Shaohuai Shi, Pengfei Xu, 2016, 2016 7th International Conference on Cloud Computing and Big Data (CCBD).

Speeding up Convolutional Neural Networks By Exploiting the Sparsity of Rectifier Units pdf

Shaohuai Shi, Xiaowen Chu, S. Shi, 2017, ArXiv.

Efficient Sparse-Dense Matrix-Matrix Multiplication on GPUs Using the Customized Sparse Storage Format pdf

Shaohuai Shi, Qiang Wang, Xiaowen Chu, 2020, 2020 IEEE 26th International Conference on Parallel and Distributed Systems (ICPADS).

Nebula-I: A General Framework for Collaboratively Training Deep Learning Models on Low-Bandwidth Cloud Clusters

S. Shi, Dianhai Yu, Peng Liu, 2022, ArXiv.

Scalable K-FAC Training for Deep Neural Networks With Distributed Preconditioning

S. Shi, Wei Wang, Lin Zhang, 2022, IEEE Transactions on Cloud Computing.

Virtual Homogeneity Learning: Defending against Data Heterogeneity in Federated Learning

S. Shi, X. Chu, Yonggang Zhang, 2022, ICML.

An Efficient Split Fine-tuning Framework for Edge and Cloud Collaborative Learning

S. Shi, X. Wang, Qing Yang, 2022, ArXiv.

Communication-Efficient Distributed Deep Learning: A Comprehensive Survey pdf

Shaohuai Shi, Zhenheng Tang, Xiaowen Chu, 2020, ArXiv.

Benchmarking Deep Learning Models and Automated Model Design for COVID-19 Detection with Chest CT Scans

X. He, X. Liu, X. Chu, 2020, medRxiv.

Performance and Power Evaluation of AI Accelerators for Training Deep Learning Models

S. Shi, Kaiyong Zhao, X. Chu, 2019, ArXiv.

Benchmarking the Performance and Energy Efficiency of AI Accelerators for AI Training

Shaohuai Shi, Zhenheng Tang, Qiang Wang, 2019, 2020 20th IEEE/ACM International Symposium on Cluster, Cloud and Internet Computing (CCGRID).

A Quantitative Survey of Communication Optimizations in Distributed Deep Learning

Shaohuai Shi, Zhenheng Tang, Xiaowen Chu, 2020, IEEE Network.

Communication-Efficient Decentralized Learning with Sparsification and Adaptive Peer Selection pdf

Shaohuai Shi, Zhenheng Tang, Xiaowen Chu, 2020, 2020 IEEE 40th International Conference on Distributed Computing Systems (ICDCS).

A Distributed Synchronous SGD Algorithm with Global Top-k Sparsification for Low Bandwidth Networks

Shaohuai Shi, Zhenheng Tang, Yuxin Wang, 2019, 2019 IEEE 39th International Conference on Distributed Computing Systems (ICDCS).

Data Resampling for Federated Learning with Non-IID Labels

S. Shi, Yiu-ming Cheung, X. Chu, 2021 .

FADNet: A Fast and Accurate Network for Disparity Estimation

Shaohuai Shi, Qiang Wang, Kaiyong Zhao, 2020, 2020 IEEE International Conference on Robotics and Automation (ICRA).

EASNet: Searching Elastic and Accurate Network Architecture for Stereo Matching

S. Shi, Kaiyong Zhao, X. Chu, 2022, ECCV.

Communication-Efficient Distributed Deep Learning with Merged Gradient Sparsification on GPUs

Shaohuai Shi, Xiaowen Chu, Yang Qin, 2020, IEEE INFOCOM 2020 - IEEE Conference on Computer Communications.

Layer-wise Adaptive Gradient Sparsification for Distributed Deep Learning with Convergence Guarantees

S. Shi, Kaiyong Zhao, X. Chu, 2019, ECAI.

Understanding Top-k Sparsification in Distributed Deep Learning pdf

Shaohuai Shi, Xiaowen Chu, Simon See, 2019, ArXiv.

MG-WFBP: Efficient Data Communication for Distributed Synchronous SGD Algorithms

Shaohuai Shi, Xiaowen Chu, S. Shi, 2018, IEEE INFOCOM 2019 - IEEE Conference on Computer Communications.

Performance Modeling and Evaluation of Distributed Deep Learning Frameworks on GPUs

Shaohuai Shi, Xiaowen Chu, S. Shi, 2017, 2018 IEEE 16th Intl Conf on Dependable, Autonomic and Secure Computing, 16th Intl Conf on Pervasive Intelligence and Computing, 4th Intl Conf on Big Data Intelligence and Computing and Cyber Science and Technology Congress(DASC/PiCom/DataCom/CyberSciTech).

A DAG Model of Synchronous Stochastic Gradient Descent in Distributed Deep Learning

Bo Li, Qiang Wang, Shaohuai Shi, 2018, 2018 IEEE 24th International Conference on Parallel and Distributed Systems (ICPADS).

A Convergence Analysis of Distributed SGD with Communication-Efficient Gradient Sparsification

Shaohuai Shi, Zhenheng Tang, Kaiyong Zhao, 2019, IJCAI.

FedML Parrot: A Scalable Federated Learning System via Heterogeneity-aware Scheduling on Sequential and Hierarchical Training

S. Shi, Chaoyang He, A. Avestimehr, 2023, ArXiv.

Supervised Learning Based Algorithm Selection for Deep Neural Networks

Shaohuai Shi, Pengfei Xu, Xiaowen Chu, 2017, 2017 IEEE 23rd International Conference on Parallel and Distributed Systems (ICPADS).

GossipFL: A Decentralized Federated Learning Framework With Sparsified and Adaptive Communication

S. Shi, X. Chu, Bo Li, 2023, IEEE Transactions on Parallel and Distributed Systems.

Towards Scalable Distributed Training of Deep Learning on Public Cloud Clusters pdf

Yongjian Wu, Yan Zhang, Bo Yang, 2020, MLSys.

Communication-Efficient Distributed Deep Learning: Survey, Evaluation, and Challenges pdf

Shaohuai Shi, Zhenheng Tang, Xiaowen Chu, 2020, ArXiv.

Evaluation and Optimization of Gradient Compression for Distributed Deep Learning

S. Shi, Bo Li, X. Chu, 2023, ArXiv.

Automated Model Design and Benchmarking of 3D Deep Learning Models for COVID-19 Detection with Chest CT Scans pdf

Shaohuai Shi, Xiaowen Chu, Shihao Wang, 2021, AAAI.

FusionAI: Decentralized Training and Deploying LLMs with Massive Consumer-Level GPUs

S. Shi, Xiaowen Chu, Kaiyong Zhao, 2023, ArXiv.

Exploiting Simultaneous Communications to Accelerate Data Parallel Distributed Deep Learning

Shaohuai Shi, Xiaowen Chu, Bo Li, 2021, IEEE INFOCOM 2021 - IEEE Conference on Computer Communications.

DeAR: Accelerating Distributed Deep Learning with Fine-Grained All-Reduce Pipelining

S. Shi, Bo Li, Wei Wang, 2023, 2302.12445.

Decoupling the All-Reduce Primitive for Accelerating Distributed Deep Learning

S. Shi, Wei Wang, X. Chu, 2023, ArXiv.

PipeMoE: Accelerating Mixture-of-Experts through Adaptive Pipelining

S. Shi, Bo Li, X. Chu, 2023, IEEE INFOCOM 2023 - IEEE Conference on Computer Communications.

Improving the Performance of Fully Connected Neural Networks by Out-of-Place Matrix Transpose pdf

Shaohuai Shi, Pengfei Xu, Xiaowen Chu, 2017, ArXiv.

LoRA-FA: Memory-efficient Low-rank Adaptation for Large Language Models Fine-tuning

S. Shi, Bo Li, X. Chu, 2023, ArXiv.

Dissecting the Runtime Performance of the Training, Fine-tuning, and Inference of Large Language Models

S. Shi, Longteng Zhang, Xinglin Pan, 2023, ArXiv.

Computer-Aided Clinical Skin Disease Diagnosis Using CNN and Object Detection Models

Shaohuai Shi, Zhenheng Tang, Xiaowen Chu, 2019, 2019 IEEE International Conference on Big Data (Big Data).

FedImpro: Measuring and Improving Client Update in Federated Learning

S. Shi, Yonggang Zhang, Zhenheng Tang, 2024, ArXiv.

Eva: A General Vectorized Approximation Framework for Second-order Optimization

S. Shi, Lin Zhang, Bo-wen Li, 2023, ArXiv.

Eva: Practical Second-order Optimization with Kronecker-vectorized Approximation

S. Shi, Lin Zhang, Bo Li, 2023, International Conference on Learning Representations.

Accelerating Distributed K-FAC with Efficient Collective Communication and Scheduling

S. Shi, Bo Li, Lin Zhang, 2023, IEEE INFOCOM 2023 - IEEE Conference on Computer Communications.

Accelerating Distributed K-FAC with Smart Parallelism of Computing and Communication Tasks

Shaohuai Shi, Bo Li, Lin Zhang, 2021, 2021 IEEE 41st International Conference on Distributed Computing Systems (ICDCS).