A Novel Neuron Connection Model Mimicking Human Beings

Neural Networks have achieved great success in many computer vision tasks, especially in image recognition. However, as neural networks grow deeper and deeper, to some extend, we've found them becoming difficult to train, and requiring samples in large scale dramatically, even with the help of Dropout and Dropconnect methods, which do improve the accuracy a bit but burdens the training process as a sacrifice. To overcome this, we propose a novel neuron connection model to generate dynamic graphs of computation. As synapses have two kinds: excitatory and inhibitory ones, our model also has two kinds of connections for neurons. In addition, we propose a training algorithm that deals with non-differentiable because the equations of the connections and activation function of neurons in our model are not really differentiable. To evaluate the effectiveness the proposed method, we apply it to the image recognition task, and the results show that our proposed model achieves state-of-the-art performance on three public datasets: MNIST, CIFAR-10, and CIFAR-100.

[1]  D. Gabor,et al.  Theory of communication. Part 1: The analysis of information , 1946 .

[2]  Luca Maria Gambardella,et al.  Deep Neural Networks Segment Neuronal Membranes in Electron Microscopy Images , 2012, NIPS.

[3]  On Reciprocal Innervation in Vaso-Motor Reflexes and the Action of Strychnine and of Chloroform thereon , 1908 .

[4]  Gray Eg.  Axo-somatic and axo-dendritic synapses of the cerebral cortex: An electron microscope study , 1959 .

[5]  J. Cowan,et al.  Excitatory and inhibitory interactions in localized populations of model neurons. , 1972, Biophysical journal.

[6]  Yann LeCun,et al.  Pedestrian Detection with Unsupervised Multi-stage Feature Learning , 2012, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[7]  Qiang Chen,et al.  Network In Network , 2013, ICLR.

[8]  E. Gray,et al.  Axo-somatic and axo-dendritic synapses of the cerebral cortex: an electron microscope study. , 1959, Journal of anatomy.

[9]  Karel Svoboda,et al.  Locally dynamic synaptic learning rules in pyramidal neuron dendrites , 2007, Nature.

[10]  Kunihiko Fukushima,et al.  Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position , 1980, Biological Cybernetics.

[11]  Geoffrey E. Hinton,et al.  Rectified Linear Units Improve Restricted Boltzmann Machines , 2010, ICML.

[12]  K. Uchizono.  Characteristics of Excitatory and Inhibitory Synapses in the Central Nervous System of the Cat , 1965, Nature.

[13]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[14]  Rob Fergus,et al.  Visualizing and Understanding Convolutional Networks , 2013, ECCV.

[15]  Luca Maria Gambardella,et al.  Flexible, High Performance Convolutional Neural Networks for Image Classification , 2011, IJCAI.

[16]  Jürgen Schmidhuber,et al.  Deep learning in neural networks: An overview , 2014, Neural Networks.

[17]  Camille Couprie,et al.  Learning Hierarchical Features for Scene Labeling , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[18]  R. Gerard.  The Interaction of Neurones , 1941 .

[19]  Lawrence D. Jackel,et al.  Backpropagation Applied to Handwritten Zip Code Recognition , 1989, Neural Computation.

[20]  Luca Maria Gambardella,et al.  Mitosis Detection in Breast Cancer Histology Images with Deep Neural Networks , 2013, MICCAI.

[21]  Jürgen Schmidhuber,et al.  Multi-column deep neural networks for image classification , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[22]  Narendra Ahuja,et al.  Cresceptron: a self-organizing neural network which grows adaptively , 1992, [Proceedings 1992] IJCNN International Joint Conference on Neural Networks.

[23]  Anil K. Jain,et al.  Artificial Neural Networks: A Tutorial , 1996, Computer.