Early experiments with neural diversity machines

The current paper introduces the concept of neural diversity machines (NDM) which, refers to hybrid artificial neural networks (HANN) with conditions on the minimum number of functions available to the network, amongst several other properties. The paper demonstrates how NDM networks can be optimized for solving different problems. The results demonstrate the feasibility of the approach and bolster some of the biological and computational arguments in favor of neural diversity. A substantial number of optimization experiments were conducted, generating a corresponding number of diverse neural architectures, which revealed several unexpected statistics, including the relative commonality of nodes combining inner-product and Gaussian functions. The paper confirms the advantages of HANN, demonstrates the potential of increasing the focus on neural diversity and hints at possible new neural computational strategies.

[1]  David E. Goldberg,et al.  Genetic Algorithms in Search Optimization and Machine Learning , 1988 .

[2]  Andrzej Bargiela,et al.  Cybernetics of Vision Systems: Toward an Understanding of Putative Functions of the Outer Retina , 2011, IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans.

[3]  Jessica A. Cardin,et al.  Neocortical Interneurons: From Diversity, Strength , 2010, Cell.

[4]  Robert E. Schapire,et al.  Theoretical Views of Boosting and Applications , 1999, ALT.

[5]  Z. Nusser.  Variability in the subcellular distribution of ion channels increases neuronal diversity , 2009, Trends in Neurosciences.

[6]  Sapiyan Baba,et al.  Unsupervised learning in second-order neural networks for motion analysis , 2011, Neurocomputing.

[7]  Antonia Azzini,et al.  Evolutionary ANNs: A state of the art survey , 2011, Intelligenza Artificiale.

[8]  Colin Giles,et al.  Learning, invariance, and generalization in high-order neural networks. , 1987, Applied optics.

[9]  Vittorio Maniezzo,et al.  Genetic evolution of the topology and weight distribution of neural networks , 1994, IEEE Trans. Neural Networks.

[10]  Pedro Antonio Gutiérrez,et al.  Hybrid Artificial Neural Networks: Models, Algorithms and Data , 2011, IWANN.

[11]  Yee Whye Teh,et al.  A Fast Learning Algorithm for Deep Belief Nets , 2006, Neural Computation.

[12]  Wlodzislaw Duch,et al.  Constructive density estimation network based on several different separable transfer functions , 2001, ESANN.

[13]  Corinna Cortes,et al.  Support-Vector Networks , 1995, Machine Learning.

[14]  P. A. Gutierrezand,et al.  Hybrid Artificial Neural Networks: Models, Algorithms and Data , 2011 .

[15]  Yoshua. Bengio,et al.  Learning Deep Architectures for AI , 2007, Found. Trends Mach. Learn..

[16]  Hak-Keung Lam,et al.  Tuning of the structure and parameters of a neural network using an improved genetic algorithm , 2003, IEEE Trans. Neural Networks.

[17]  Pedro Antonio Gutiérrez,et al.  Combined projection and kernel basis functions for classification in evolutionary neural networks , 2009, Neurocomputing.

[18]  P. Somogyi,et al.  Neuronal Diversity and Temporal Dynamics: The Unity of Hippocampal Circuit Operations , 2008, Science.

[19]  Nathan Intrator,et al.  A Hybrid Projection Based and Radial Basis Function Architecture , 2000, Multiple Classifier Systems.

[20]  Paul Nurse,et al.  Cell Division Intersects with Cell Geometry , 2010, Cell.

[21]  G. Buzsáki,et al.  Interneuron Diversity series: Circuit complexity and axon wiring economy of cortical interneurons , 2004, Trends in Neurosciences.

[22]  Andrew B. Kahng,et al.  Variability , 2002, IEEE Des. Test Comput..

[23]  Kurt Hornik,et al.  Multilayer feedforward networks are universal approximators , 1989, Neural Networks.

[24]  Vladimir Vapnik,et al.  A new learning paradigm: Learning using privileged information , 2009, Neural Networks.

[25]  Rainer Storn,et al.  Differential Evolution – A Simple and Efficient Heuristic for global Optimization over Continuous Spaces , 1997, J. Glob. Optim..

[26]  E. Marder,et al.  Variability, compensation and homeostasis in neuron and network function , 2006, Nature Reviews Neuroscience.

[27]  Stefan Wermter,et al.  Hybrid neural systems: from simple coupling to fully integrated neural networks , 1999 .

[28]  W. Pitts,et al.  A Logical Calculus of the Ideas Immanent in Nervous Activity (1943) , 2021, Ideas That Created the Future.

[29]  I. Soltesz.  Diversity in the Neuronal Machine: Order and Variability in Interneuronal Microcircuits , 2005 .

[30]  Peter Secretan.  Learning , 1965, Mental Health.

[31]  Wlodzislaw Duch,et al.  Optimal transfer function neural networks , 2001, ESANN.

[32]  R. Masland.  Neuronal diversity in the retina , 2001, Current Opinion in Neurobiology.