ImageNet Training by CPU: AlexNet in 11 Minutes and ResNet-50 in 48 Minutes

Since its creation, the ImageNet 1-k benchmark set has played a significant role as a benchmark for ascertaining the accuracy of different deep neural net (DNN) models on the classification problem. Moreover, in recent years it has also served as the principal benchmark for assessing different approaches to DNN training. Finishing a 90-epoch ImageNet1k training with ResNet-50 on a NVIDIA M40 GPU takes 14 days. This training requires 10 single precision operations in total. On the other hand, the world’s current fastest supercomputer can finish 2× 10 single precision operations per second. If we can make full use of the computing capability of a supercomputer for DNN training, we should be able to finish the 90-epoch ResNet-50 training in five seconds. Over the last two years a number of researchers have focused on how to close this significant performance gap through scaling DNN training to larger numbers of processors. Most successful approaches to scaling the training of ImageNet have used the approach of synchronous stochastic gradient descent. However, to scale synchronous stochastic gradient descent one must also increase the batch size used in each iteration. Thus, for many researchers, the focus on scaling DNN training has translated into a focus on developing approaches that enable increasing the batch size in data-parallel synchronous stochastic gradient descent without losing accuracy over a fixed number of epochs. As a result, we have seen the batch size and number of processors successfully utilized increase from 1K batch/128 processors to 8K batch/256 processors over the last two years. The recently published LARS algorithm increased batch size further to 32K for some DNN models. Following up on this work, we wished to confirm that LARS could be used to further scale the number of processors efficiently used in DNN training and, and as a result, further reduce the total training time. In this paper we present the results of this investigation: using LARS we were able to efficiently utilize 1024 CPUs to finish the 100-epoch ImageNet training with AlexNet in 11 minutes, and we finished 90-epoch ImageNet training with ResNet-50 in 48 minutes (batch size = 32K). Furthermore, when we increase the batch size to above 20K, our accuracy is much higher than Facebook’s on corresponding batch sizes (Figure 1). Our source code is available upon request. It will also be released in Intel Caffe. Figure 1: Because we use weaker data augmentation, our baseline’s accuracy is slightly lower than Facebook’s version (76.2% vs 75.4%). However, at very large batch sizes our accuracy is much higher than Facebook’s accuracy. Facebook’s accuracy is from their own report (Goyal et al. 2017). Our accuracy scaling efficiency is much higher than Facebook’s version