Non-Euclidean Vector Product for Neural Networks

We present a non-Euclidean vector product for artificial neural networks. The vector product operator does not require any multiplications while providing correlation information between two vectors. Ordinary neurons require inner product of two vectors. We propose a class of neural networks with the universal approximation property over the space of Lebesgue integrable functions based on the proposed non-Euclidean vector product. In this new network, the “product” of two real numbers is defined as the sum of their absolute values, with the sign determined by the sign of the product of the numbers. This “product” is used to construct a vector product in RN. The vector product induces the l1 norm. The additive neural network successfully solves the XOR problem. Experiments on MNIST and CIFAR datasets show that the classification performance of the proposed additive neural network is comparable to the corresponding multi-layer perceptron and convolutional neural networks.

[1]  Yoshua Bengio,et al.  BinaryConnect: Training Deep Neural Networks with binary weights during propagations , 2015, NIPS.

[2]  Patrice Y. Simard,et al.  Best practices for convolutional neural networks applied to visual document analysis , 2003, Seventh International Conference on Document Analysis and Recognition, 2003. Proceedings..

[3]  Jim D. Garside,et al.  SpiNNaker: A 1-W 18-Core System-on-Chip for Massively-Parallel Neural Network Simulation , 2013, IEEE Journal of Solid-State Circuits.

[4]  Jongkil Park,et al.  A 65k-neuron 73-Mevents/s 22-pJ/event asynchronous micro-pipelined integrate-and-fire array transceiver , 2014, 2014 IEEE Biomedical Circuits and Systems Conference (BioCAS) Proceedings.

[5]  Farida Cheriet,et al.  Accurate and Efficient Hyperbolic Tangent Activation Function on FPGA using the DCT Interpolation Filter , 2016, ArXiv.

[6]  A. Enis Çetin,et al.  Co-difference based object tracking algorithm for infrared videos , 2016, 2016 IEEE International Conference on Image Processing (ICIP).

[7]  Martín Abadi,et al.  TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems , 2016, ArXiv.

[8]  A. Enis Çetin,et al.  Energy efficient cosine similarity measures according to a convex cost function , 2016, Signal, Image and Video Processing.

[9]  Aysegul Uner,et al.  Multiplication-free Neural Networks , 2015, 2015 23nd Signal Processing and Communications Applications Conference (SIU).

[10]  G. Lewicki,et al.  Approximation by Superpositions of a Sigmoidal Function , 2003 .

[11]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[12]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[13]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[14]  Geoffrey E. Hinton,et al.  Learning representations by back-propagating errors , 1986, Nature.

[15]  Song Han,et al.  Learning both Weights and Connections for Efficient Neural Network , 2015, NIPS.

[16]  A. Enis Çetin,et al.  A multiplication-free framework for signal processing and applications in biomedical image analysis , 2013, 2013 IEEE International Conference on Acoustics, Speech and Signal Processing.

[17]  Jürgen Schmidhuber,et al.  A Machine Learning Approach to Visual Perception of Forest Trails for Mobile Robots , 2016, IEEE Robotics and Automation Letters.

[18]  Andrew S. Cassidy,et al.  Convolutional networks for fast, energy-efficient neuromorphic computing , 2016, Proceedings of the National Academy of Sciences.

[19]  Johannes Schemmel,et al.  Six Networks on a Universal Neuromorphic Computing Substrate , 2012, Front. Neurosci..

[20]  Anthony J. Calise,et al.  Nonlinear adaptive flight control using neural networks , 1998 .

[21]  Ali Farhadi,et al.  XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks , 2016, ECCV.

[22]  A. Enis Çetin,et al.  Image Description Using a Multiplier-Less Operator , 2009, IEEE Signal Processing Letters.

[23]  Giacomo Indiveri,et al.  An Event-Based Neural Network Architecture With an Asynchronous Programmable Synaptic Memory , 2014, IEEE Transactions on Biomedical Circuits and Systems.

[24]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[25]  Ran El-Yaniv,et al.  Binarized Neural Networks , 2016, ArXiv.