Iteration number-based hierarchical gradient aggregation for distributed deep learning

Distributed deep learning can effectively accelerate neural model training, which employs multiple workers at a cluster of nodes to train a neural network in a parallel way. In this paper, we propose InHAD, an asynchronous distributed deep learning protocol, whose key novelty lies in the design of hierarchical gradient communication and aggregation. The local aggregation is conducted inside a computing node to aggregate the gradients produced by local workers, while global aggregation is conducted at the parameter server to calculate new model parameters based on results of local aggregations. An iteration number-based mechanism is designed to guarantee the convergence of the training. Worker nodes keep sending IN to parameter server and the latter decides when to pull gradients from workers by counting IN and updates global model weights/parameters. With the IN-based hierarchical aggregation technique, InHAD can save communication cost by reducing the number of gradients transferred and speed up the convergence by limiting the staleness of gradients. We conduct extensive experiments at the Tianhe-2 supercomputer system to evaluate the performance of InHAD. Two neural networks are trained on two classical datasets, and similar protocols like Horovod and ASP are tested for comparisons. The results show that InHAD can achieve much higher acceleration than ASP and nearly the same accuracy as Horovod .