Comparing Deep and Dendrite Neural Networks: A Case Study

In this paper, a comparative study between two different neural network models is performed for a very simple type of classificaction problem in 2D. The first model is a deep neural network and the second is a dendrite morphological neuron. The metrics to be compared are: training time, classification accuracies and number of learning parameters. We also compare the decision boundaries generated by both models. The experiments show that the dendrite morphological neurons surpass the deep neural networks by a wide margin in terms of higher accuracies and a lesser number of parameters. From this, we raise the hypothesis that deep learning networks can be improved adding morphological neurons.

[1]  V. Litovski,et al.  Annealing based dynamic learning in second-order neural networks , 1996, Proceedings of International Conference on Neural Networks (ICNN'96).

[2]  David J. C. MacKay,et al.  A Practical Bayesian Framework for Backpropagation Networks , 1992, Neural Computation.

[3]  Yoshua Bengio,et al.  Deep Sparse Rectifier Neural Networks , 2011, AISTATS.

[4]  Gerhard X. Ritter,et al.  Lattice algebra approach to single-neuron computation , 2003, IEEE Trans. Neural Networks.

[5]  Colin Giles,et al.  Learning, invariance, and generalization in high-order neural networks. , 1987, Applied optics.

[6]  Yoshua Bengio,et al.  Learning long-term dependencies with gradient descent is difficult , 1994, IEEE Trans. Neural Networks.

[7]  A. G. Ivakhnenko,et al.  Polynomial Theory of Complex Systems , 1971, IEEE Trans. Syst. Man Cybern..

[8]  Juan Humberto Sossa Azuela,et al.  Dendrite morphological neurons trained by stochastic gradient descent , 2016, SSCI.

[9]  Juan Humberto Sossa Azuela,et al.  Efficient training for dendrite morphological neural networks , 2014, Neurocomputing.

[10]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[11]  Gerhard X. Ritter,et al.  Learning In Lattice Neural Networks that Employ Dendritic Computing , 2006, FUZZ-IEEE.

[12]  James L. McClelland,et al.  Parallel distributed processing: explorations in the microstructure of cognition, vol. 1: foundations , 1986 .

[13]  Amir Hussain.  A new neural network structure for temporal signal processing , 1997, 1997 IEEE International Conference on Acoustics, Speech, and Signal Processing.

[14]  Geoffrey E. Hinton,et al.  Deep Learning , 2015, Nature.

[15]  Petros Maragos,et al.  Neural networks with hybrid morphological/rank/linear nodes and their application to handwritten character recognition , 1998, 9th European Signal Processing Conference (EUSIPCO 1998).

[16]  Philip D. Wasserman,et al.  Neural networks. II. What are they and why is everybody so interested in them now? , 1988, IEEE Expert.

[17]  Jürgen Schmidhuber,et al.  Deep learning in neural networks: An overview , 2014, Neural Networks.

[18]  Kevin N. Gurney,et al.  Training nets of hardware realizable sigma-pi units , 1992, Neural Networks.

[19]  David E. Rumelhart,et al.  Product Units: A Computationally Powerful and Biologically Plausible Extension to Backpropagation Networks , 1989, Neural Computation.

[20]  Tara N. Sainath,et al.  Deep Neural Networks for Acoustic Modeling in Speech Recognition: The Shared Views of Four Research Groups , 2012, IEEE Signal Processing Magazine.

[21]  Gerhard X. Ritter,et al.  Morphological perceptrons with dendritic structure , 2003, The 12th IEEE International Conference on Fuzzy Systems, 2003. FUZZ '03..

[22]  C. V. D. Malsburg,et al.  Frank Rosenblatt: Principles of Neurodynamics: Perceptrons and the Theory of Brain Mechanisms , 1986 .

[23]  Qinghua Zhang,et al.  Wavelet networks , 1992, IEEE Trans. Neural Networks.

[24]  Samy Bengio,et al.  The Handbook of Brain Theory and Neural Networks , 2002 .

[25]  Corinna Cortes,et al.  Support-Vector Networks , 1995, Machine Learning.

[26]  Yoshua. Bengio,et al.  Learning Deep Architectures for AI , 2007, Found. Trends Mach. Learn..