Attentiondrop for Convolutional Neural Networks

Dropout has been widely used in fully connected networks but becomes less effective for convolutional neural networks (CNNs), since the spatially correlated features still allow dropped information to flow through the network. To make dropout more practical for CNNs, structured dropout methods have been recently proposed by dropping regions with fixed shapes and random positions, which nonetheless may lead to unexpected discarding of information. To address this problem, in this paper, we propose a novel dropout variant based on attention information named AttentionDrop that drops features adaptively. Specifically, it precisely localizes masks that have irregular shapes according to the values of activation units. In addition, the use of soft values in adaptive masks lowers the risk of a complete loss of indispensable information. Experimental results demonstrate the effectiveness of our AttentionDrop on public datasets for image classification.