Effect of Absolute Cosine Value Regularization on VGG-19

The search for a more accurate Convolutional Neural Network (CNN) is a continuous process of unlimited possibilities. In order to simplify this process, the effects of algorithms on a network must be examined individually in order to determine their merits. The algorithm examined by this paper is a filter weight matrix regularizer designed to promote optimal information distillation. Absolute Cosine Value Regularization (ACVR) is a regularization technique hypothesized to increase the representational power of CNNs by using a Gradient Descent Orthogonalization algorithm to force the vectors that constitute their filters at any given convolutional layer to occupy unique positions in $\mathbb{R}^{n}$. This method has previously been given a mathematical definition, implementation description, and in addition, has been demonstrated to be effective at producing high diversity filter vectors in $\mathbb{R}^{3}$. However, the effect of this Regularizer on a full-scale CNN architecture has yet to be fully examined. This paper aims to determine the merits of this Regularizer by presenting experimental results generated by training the well-established CNN architecture VGG-19 with, and without its presence, using the CIFAR-10 image classification data set. This paper then goes on to propose the Dynamic-ACVR (D-ACVR) algorithm, demonstrating that at optimal configurations, this Regularization scheme can increase network accuracy by up to 3.12%.

[1]  Lixin Fan,et al.  Simultaneously Learning Architectures and Features of Deep Neural Networks , 2019, ICANN.

[2]  Yang Yu,et al.  Diversity Regularized Machine , 2011, IJCAI.

[3]  Alex Krizhevsky,et al.  Learning Multiple Layers of Features from Tiny Images , 2009 .

[4]  Ali Farhadi,et al.  You Only Look Once: Unified, Real-Time Object Detection , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[5]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[6]  Mohamed El-Sharkawy,et al.  Increasing CNN Representational Power Using Absolute Cosine Value Regularization , 2020, 2020 IEEE 63rd International Midwest Symposium on Circuits and Systems (MWSCAS).

[7]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[8]  Jianfeng Zhan,et al.  Cosine Normalization: Using Cosine Similarity Instead of Dot Product in Neural Networks , 2017, ICANN.

[9]  Chiman Kwan,et al.  Deep Learning Based Target Tracking and Classification for Infrared Videos Using Compressive Measurements , 2019, Journal of Signal and Information Processing.

[10]  Yann LeCun,et al.  Optimal Brain Damage , 1989, NIPS.

[11]  Bo Chen,et al.  MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications , 2017, ArXiv.

[12]  Hanan Samet,et al.  Pruning Filters for Efficient ConvNets , 2016, ICLR.

[13]  Raghuraman Krishnamoorthi,et al.  Quantizing deep convolutional networks for efficient inference: A whitepaper , 2018, ArXiv.

[14]  Joel Nothman,et al.  SciPy 1.0-Fundamental Algorithms for Scientific Computing in Python , 2019, ArXiv.

[15]  Ralph Etienne-Cummings,et al.  Detection and Confirmation of Multiple Human Targets Using Pixel-Wise Code Aperture Measurements , 2020, J. Imaging.