SBNN: Slimming binarized neural network

Abstract With the rapid developments of deep neural networks related applications, approaches for accelerating computationally intensive convolutional neural networks, such as network quantization, pruning, knowledge distillation, have attracted ever-increasing attention. Network binarization is an extreme form of network quantization technique, which binarizes the network weights and/or activation values to save computational resources. However, it often introduces noises into the network, and requires larger model size (more parameters) to compensate for the loss of representation capacity. To address the model complexity reduction challenges and further improve the network performance, this paper proposes an approach: slimming binarized neural networks (SBNN), which reduces complexity of binarized networks with acceptable accuracy loss. SBNN prunes the convolutional layers and fully-connected layer in a binarized network. Then it is refined by the proposed SoftSign function, knowledge distillation and full-precision computation to enhance the network accuracy. The proposed SBNN can be also conveniently applied to a pre-trained binarized network. We demonstrate the effectiveness of our approach through several state-of-the-art binarized models. For AlexNet and ResNet-18 on ILSVRC-2012 dataset, SBNN obtains negligible accuracy loss but even a better accuracy than the pre-pruning model while using only 75% of original filters.