Artificial Neural Network Learning: A Comparative Review

Various neural learning procedures have been proposed by different researchers in order to adapt suitable controllable parameters of neural network architectures. These can be from simple Hebbian procedures to complicated algorithms applied to individual neurons or assemblies in a neural structure. The paper presents an organized review of various learning techniques, classified according to basic characteristics such as chronology, applicability, functionality, stochasticity etc. Some of the learning procedures that have been used for the training of generic and specific neural structures, and will be reviewed are: Hebbian-like (Grossberg, Sejnowski, Sutton, Bienenstock, Oja & Karhunen, Sanger, Yuile et al., Hasselmo, Kosko, Cheung & Omidvar), Reinforcement learning, Min-max learning, Stochastic learning, Genetics-based learning, Artificial life-based learning. The various learning procedures will be critically compared, and future trends will be highlighted.

[1]  Michael E. Hasselmo,et al.  Runaway synaptic modification in models of cortex: Implications for Alzheimer's disease , 1994, Neural Networks.

[2]  Kwok-Wo Wong,et al.  A pruning method for the recursive least squared algorithm , 2001, Neural Networks.

[3]  James L. McClelland,et al.  Parallel distributed processing: explorations in the microstructure of cognition, vol. 1: foundations , 1986 .

[4]  Tarun Khanna,et al.  Foundations of neural networks , 1990 .

[5]  Bernard Widrow,et al.  Punish/Reward: Learning with a Critic in Adaptive Threshold Systems , 1973, IEEE Trans. Syst. Man Cybern..

[6]  Robert J. Marks,et al.  Neural Smithing: Supervised Learning in Feedforward Artificial Neural Networks , 1999 .

[7]  Patrick K. Simpson,et al.  Neural Networks Theory, Technology and Applications , 1995 .

[8]  S. Mitter,et al.  Recursive stochastic algorithms for global optimization in R d , 1991 .

[9]  Yoav Freund,et al.  Boosting a weak learning algorithm by majority , 1995, COLT '90.

[10]  C. D. Gelatt,et al.  Optimization by Simulated Annealing , 1983, Science.

[11]  Bernard Widrow,et al.  Adaptive switching circuits , 1988 .

[12]  F. Aluffi-Pentini,et al.  Global optimization and stochastic differential equations , 1985 .

[13]  P. K. Simpson,et al.  Fuzzy min-max neural networks , 1991, [Proceedings] 1991 IEEE International Joint Conference on Neural Networks.

[14]  B. Kosko,et al.  Feedback stability and unsupervised learning , 1988, IEEE 1988 International Conference on Neural Networks.

[15]  Simon Haykin,et al.  Neural Networks: A Comprehensive Foundation , 1998 .

[16]  E. Bienenstock,et al.  Theory for the development of neuron selectivity: orientation specificity and binocular interaction in visual cortex , 1982, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[17]  Andrzej Cichocki,et al.  Neural networks for optimization and signal processing , 1993 .

[18]  Jirí Benes,et al.  On neural networks , 1990, Kybernetika.

[19]  S Grossberg,et al.  Some nonlinear networks capable of learning a spatial pattern of arbitrary complexity. , 1968, Proceedings of the National Academy of Sciences of the United States of America.

[20]  Bart Kosko,et al.  Neural networks and fuzzy systems , 1998 .

[21]  Bart Kosko,et al.  Unsupervised learning in noise , 1990, International 1989 Joint Conference on Neural Networks.

[22]  Terence D. Sanger,et al.  Optimal unsupervised learning in a single-layer linear feedforward neural network , 1989, Neural Networks.

[23]  Wei-Der Chang,et al.  A feedforward neural network with function shape autotuning , 1996, Neural Networks.

[24]  Erkki Oja,et al.  Principal components, minor components, and linear neural networks , 1992, Neural Networks.

[25]  Carsten Peterson,et al.  A Mean Field Theory Learning Algorithm for Neural Networks , 1987, Complex Syst..

[26]  H. Szu.  Fast simulated annealing , 1987 .

[27]  Patrick K. Simpson.  Fuzzy min-max classification with neural networks , 1991, [1991 Proceedings] IEEE Conference on Neural Networks for Ocean Engineering.

[28]  E. Oja,et al.  On stochastic approximation of the eigenvectors and eigenvalues of the expectation of a random matrix , 1985 .

[29]  A G Barto,et al.  Learning by statistical cooperation of self-interested neuron-like computing elements. , 1985, Human neurobiology.

[30]  Erkki Oja,et al.  Principal component analysis by homogeneous neural networks, part II: Analysis and extentions of the learning algorithm , 1992 .

[31]  Edmondo Trentin,et al.  Networks with trainable amplitude of activation functions , 2001, Neural Networks.

[32]  John Y. Cheung,et al.  Mathematical Analysis of Learning Behavior of Neuronal Models , 1987, NIPS.

[33]  A. Harry Klopf,et al.  A drive-reinforcement model of single neuron function , 1987 .

[34]  Edgar Sanchez-Sinencio,et al.  Artificial Neural Networks: Paradigms, Applications, and Hardware Implementations , 1994 .

[35]  Javier R. Movellan,et al.  Benefits of gain: speeded learning and minimal hidden layers in back-propagation networks , 1991, IEEE Trans. Syst. Man Cybern..

[36]  D. M. Kammen,et al.  Quadrature and the development of orientation selective cortical cells by Hebb rules , 1989, Biological Cybernetics.

[37]  Geoffrey E. Hinton,et al.  A Learning Algorithm for Boltzmann Machines , 1985, Cogn. Sci..

[38]  Herbert A. Simon,et al.  The Sciences of the Artificial , 1970 .

[39]  E. Oja.  Simplified neuron model as a principal component analyzer , 1982, Journal of mathematical biology.

[40]  Andries Petrus Engelbrecht,et al.  Cooperative learning in neural networks using particle swarm optimizers , 2000, South Afr. Comput. J..

[41]  A G Barto,et al.  Toward a modern theory of adaptive networks: expectation and prediction. , 1981, Psychological review.

[42]  P. Werbos,et al.  Beyond Regression : "New Tools for Prediction and Analysis in the Behavioral Sciences , 1974 .

[43]  Teuvo Kohonen,et al.  Self-organization and associative memory: 3rd edition , 1989 .

[44]  M.H. Hassoun,et al.  Fundamentals of Artificial Neural Networks , 1996, Proceedings of the IEEE.

[45]  J. Leo van Hemmen,et al.  Universality of unlearning , 1994, Neural Networks.

[46]  Richard S. Sutton,et al.  Neuronlike adaptive elements that can solve difficult learning control problems , 1983, IEEE Transactions on Systems, Man, and Cybernetics.

[47]  S. Kaplan.  The Physiology of Thought , 1950 .

[48]  J. J. Hopfield,et al.  ‘Unlearning’ has a stabilizing effect in collective memories , 1983, Nature.

[49]  Robert B. Ash,et al.  Information Theory , 2020, The SAGE International Encyclopedia of Mass Media and Society.

[50]  Constantinos S. Pattichis,et al.  A biologically inspired neural network composed of dissimilar single neuron models , 2001, 2001 Conference Proceedings of the 23rd Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[51]  B. Kosco.  Differential Hebbian learning , 1987 .

[52]  T. Sejnowski.  Statistical constraints on synaptic plasticity. , 1977, Journal of theoretical biology.

[53]  James Kennedy,et al.  Particle swarm optimization , 2002, Proceedings of ICNN'95 - International Conference on Neural Networks.