Large Margin Deep Neural Networks: Theory and Algorithms

Deep neural networks (DNN) have achieved huge practical suc cess in recent years. However, its theoretical properties (in particular genera lization ability) are not yet very clear, since existing error bounds for neural networks cannot be directly used to explain the statistical behaviors of practically adopte d DNN models (which are multi-class in their nature and may contain convolutional l ayers). To tackle the challenge, we derive a new margin bound for DNN in this paper, in which the expected0-1 error of a DNN model is upper bounded by its empirical margin e rror plus a Rademacher Average based capacity term. This new boun d is very general and is consistent with the empirical behaviors of DNN models ob erved in our experiments. According to the new bound, minimizing the emp irical margin error can effectively improve the test performance of DNN. We ther efore propose large margin DNN algorithms, which impose margin penalty terms to the cross entropy loss of DNN, so as to reduce the margin error during the traini ng process. Experimental results show that the proposed algorithms can achiev e s gnificantly smaller empirical margin errors, as well as better test performance s than the standard DNN algorithm.