Learning in the feed-forward random neural network: A critical review

The Random Neural Network (RNN) has received, since its inception in 1989, considerable attention and has been successfully used in a number of applications. In this critical review paper we focus on the feed-forward RNN model and its ability to solve classification problems. In particular, we paid special attention to the RNN literature related with learning algorithms that discover the RNN interconnection weights, suggested other potential algorithms that can be used to find the RNN interconnection weights, and compared the RNN model with other neural-network based and non-neural network based classifier models. In review, the extensive literature review and experimentation with the RNN feed-forward model provided us with the necessary guidance to introduce six critical review comments that identify some gaps in the RNN's related literature and suggest directions for future research.

[1]  Jooyoung Park,et al.  Universal Approximation Using Radial-Basis-Function Networks , 1991, Neural Computation.

[2]  Jesús Cid-Sueiro,et al.  Cost functions to estimate a posteriori probabilities in multiclass problems , 1999, IEEE Trans. Neural Networks.

[3]  Zwe-Lee Gaing,et al.  Particle swarm optimization to solving the economic dispatch considering the generator constraints , 2003 .

[4]  Don R. Hush,et al.  Error surfaces for multilayer perceptrons , 1992, IEEE Trans. Syst. Man Cybern..

[5]  Yue Shi,et al.  A modified particle swarm optimizer , 1998, 1998 IEEE International Conference on Evolutionary Computation Proceedings. IEEE World Congress on Computational Intelligence (Cat. No.98TH8360).

[6]  Klaus-Robert Müller,et al.  Efficient BackProp , 2012, Neural Networks: Tricks of the Trade.

[7]  Said E. El-Khamy,et al.  Random neural network filter for land mine detection , 1999, Proceedings of the Sixteenth National Radio Science Conference. NRSC'99 (IEEE Cat. No.99EX249).

[8]  Terence Soule,et al.  Comparison of Genetic Algorithm and Particle Swarm Optimizer When Evolving a Recurrent Neural Network , 2003, GECCO.

[9]  William H. Press,et al.  Numerical recipes in C , 2002 .

[10]  John C. Platt,et al.  Fast training of support vector machines using sequential minimal optimization, advances in kernel methods , 1999 .

[11]  Andreas Stafylopatis,et al.  Training the random neural network using quasi-Newton methods , 2000, Eur. J. Oper. Res..

[12]  Erol Gelenbe,et al.  Wafer surface reconstruction from top–down scanning electron microscope images , 2004 .

[13]  Pierre Tirilly,et al.  Evaluating Users' Satisfaction in Packet Networks Using Random Neural Networks , 2006, ICANN.

[14]  F. Aluffi-Pentini,et al.  Global optimization and stochastic differential equations , 1985 .

[15]  Erol Gelenbe,et al.  Random network learning and image compression , 1994, Proceedings of 1994 IEEE International Conference on Neural Networks (ICNN'94).

[16]  Gerardo Rubino,et al.  Controlling Multimedia QoS in the Future Home Network Using the PSQA Metric , 2006, Comput. J..

[17]  Erol Gelenbe,et al.  Neural network methods for volumetric magnetic resonance imaging of the human brain , 1996 .

[18]  Shangxu Hu,et al.  A New Approach to Improve Particle Swarm Optimization , 2003, GECCO.

[19]  José Aguilar,et al.  Resolution of pattern recognition problems using a hybrid Genetic/Random Neural Network learning algorithm , 2005, Pattern Analysis and Applications.

[20]  Michael Georgiopoulos,et al.  Properties of learning related to pattern diversity in ART1 , 1991, Neural Networks.

[21]  Erol Gelenbe,et al.  Feature-based RNN target recognition , 1998, Defense, Security, and Sensing.

[22]  X. Xu,et al.  A case study of solving optimization problems using neural networks , 1988, Neural Networks.

[23]  Geoffrey E. Hinton,et al.  Learning internal representations by error propagation , 1986 .

[24]  Georgia Sakellari,et al.  The Cognitive Packet Network: A Survey , 2010, Comput. J..

[25]  Michael Georgiopoulos,et al.  Convergence Properties of Learning in ART1 , 1990, Neural Computation.

[26]  M. A. Abido.  Optimal des'ign of Power System Stabilizers Using Particle Swarm Opt'imization , 2002, IEEE Power Engineering Review.

[27]  Catherine Blake,et al.  UCI Repository of machine learning databases , 1998 .

[28]  Ugur Halici,et al.  Reinforcement learning with internal expectation for the random neural network , 2000, Eur. J. Oper. Res..

[29]  Erol Gelenbe,et al.  Detecting Denial of Service Attacks with Bayesian Classifiers and the Random Neural Network , 2007, 2007 IEEE International Fuzzy Systems Conference.

[30]  Michael Georgiopoulos,et al.  Properties of learning in ARTMAP , 1994, Neural Networks.

[31]  Erol Gelenbe,et al.  Video quality and traffic QoS in learning-based subsampled and receiver-interpolated video sequences , 2000, IEEE Journal on Selected Areas in Communications.

[32]  Heinz Mühlenbein,et al.  Predictive Models for the Breeder Genetic Algorithm I. Continuous Parameter Optimization , 1993, Evolutionary Computation.

[33]  Cheng-Yan Kao,et al.  A Robust Evolutionary Algorithm for Training Neural Networks , 2001, Neural Computing & Applications.

[34]  N. B. Karayiannis.  ALADIN: algorithms for Learning and Architecture DetermINation , 1994 .

[35]  Zhi-Hong Mao,et al.  Function approximation with spiked random networks , 1999, IEEE Trans. Neural Networks.

[36]  M. A. Abido,et al.  Optimal power flow using particle swarm optimization , 2002 .

[37]  V. Nedeljkovic,et al.  A novel multilayer neural networks training algorithm that minimizes the probability of classification error , 1992, Proceedings., 11th IAPR International Conference on Pattern Recognition. Vol.II. Conference B: Pattern Recognition Methodology and Systems.

[38]  Huihe Shao,et al.  An ANN's evolved by a new evolutionary system and its application , 2000, Proceedings of the 39th IEEE Conference on Decision and Control (Cat. No.00CH37187).

[39]  Teresa Bernarda Ludermir,et al.  Particle Swarm Optimization of Neural Network Architectures andWeights , 2007, 7th International Conference on Hybrid Intelligent Systems (HIS 2007).

[40]  Stephen Grossberg,et al.  Adaptive pattern classification and universal recoding: II. Feedback, expectation, olfaction, illusions , 1976, Biological Cybernetics.

[41]  Gulay Oke,et al.  Likelihood ratios and recurrent random neural networks in detection of denial of service attacks , 2007 .

[42]  Etienne Barnard,et al.  Avoiding false local minima by proper initialization of connections , 1992, IEEE Trans. Neural Networks.

[43]  Yan-Da Li,et al.  Approximation by random networks with bounded number of layers , 1999, Neural Networks for Signal Processing IX: Proceedings of the 1999 IEEE Signal Processing Society Workshop (Cat. No.98TH8468).

[44]  D. E. Goldberg,et al.  Genetic Algorithms in Search , 1989 .

[45]  James L. McClelland,et al.  Parallel distributed processing: explorations in the microstructure of cognition, vol. 1: foundations , 1986 .

[46]  Heekuck Oh,et al.  Neural Networks for Pattern Recognition , 1993, Adv. Comput..

[47]  S. Katagiri,et al.  Discriminative Learning for Minimum Error Classification , 2009 .

[48]  S. García,et al.  An Extension on "Statistical Comparisons of Classifiers over Multiple Data Sets" for all Pairwise Comparisons , 2008 .

[49]  Ge Xiurun,et al.  An improved PSO-based ANN with simulated annealing technique , 2005, Neurocomputing.

[50]  Taskin Koçak,et al.  Survey of random neural network applications , 2000, Eur. J. Oper. Res..

[51]  Erol Gelenbe,et al.  Low bit-rate video compression with neural networks and temporal subsampling , 1996, Proc. IEEE.

[52]  Erol Gelenbe,et al.  Video compression with random neural networks , 1996, Proceedings of International Workshop on Neural Networks for Identification, Control, Robotics and Signal/Image Processing.

[53]  D. R. Hush,et al.  Error surfaces for multi-layer perceptrons , 1991, IJCNN-91-Seattle International Joint Conference on Neural Networks.

[54]  Gerardo Rubino,et al.  Real-Time Video Quality Assessment in Packet Networks: A Neural Network Model , 2001 .

[55]  Erol Gelenbe,et al.  Learning in the Recurrent Random Neural Network , 1992, Neural Computation.

[56]  Georgios Loukas,et al.  A Denial of Service Detector based on Maximum Likelihood Detection and the Random Neural Network , 2007, Comput. J..

[57]  Erol Gelenbe,et al.  Autonomous smart routing for network QoS , 2004 .

[58]  G. J. Gibson,et al.  On the decision regions of multilayer perceptrons , 1990, Proc. IEEE.

[59]  E. Cantu-Paz,et al.  An empirical comparison of combinations of evolutionary algorithms and neural networks for classification problems , 2005, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[60]  Brian D. Ripley,et al.  Neural Networks and Related Methods for Classification , 1994 .

[61]  James Kennedy,et al.  Particle swarm optimization , 2002, Proceedings of ICNN'95 - International Conference on Neural Networks.

[62]  Gerardo Rubino,et al.  Levenberg - Marquardt Training Algorithms for Random Neural Networks , 2011, Comput. J..

[63]  Erol Gelenbe,et al.  Image and video compression , 1998 .

[64]  J. Salerno,et al.  Using the particle swarm optimization technique to train a recurrent neural model , 1997, Proceedings Ninth IEEE International Conference on Tools with Artificial Intelligence.

[65]  Yaochu Jin,et al.  Pareto-based Multi-Objective Machine Learning , 2007, 7th International Conference on Hybrid Intelligent Systems (HIS 2007).

[66]  Erol Gelenbe,et al.  Image Enhacement and Fusion with the Random Neural Network , 1997, Turkish Journal of Electrical Engineering and Computer Sciences.

[67]  Wei-Yin Loh,et al.  A Comparison of Prediction Accuracy, Complexity, and Training Time of Thirty-Three Old and New Classification Algorithms , 2000, Machine Learning.

[68]  Stephen Grossberg,et al.  The ART of adaptive pattern recognition by a self-organizing neural network , 1988, Computer.

[69]  Georgios C. Anagnostopoulos,et al.  Exemplar-based pattern recognition via semi-supervised learning , 2003, Proceedings of the International Joint Conference on Neural Networks, 2003..

[70]  R. Raghavan,et al.  Gradient descent fails to separate , 1988, IEEE 1988 International Conference on Neural Networks.

[71]  Bernhard Sendhoff,et al.  Evolutionary Multi-objective Optimization of Spiking Neural Networks , 2007, ICANN.

[72]  J. Dennis,et al.  A closer look at drawbacks of minimizing weighted sums of objectives for Pareto set generation in multicriteria optimization problems , 1997 .

[73]  Janez Demsar,et al.  Statistical Comparisons of Classifiers over Multiple Data Sets , 2006, J. Mach. Learn. Res..

[74]  Georgios C. Anagnostopoulos,et al.  Category regions as new geometrical concepts in Fuzzy-ART and Fuzzy-ARTMAP , 2002, Neural Networks.

[75]  Martin A. Riedmiller,et al.  A direct adaptive method for faster backpropagation learning: the RPROP algorithm , 1993, IEEE International Conference on Neural Networks.

[76]  Bernhard Sendhoff,et al.  Generalization Improvement in Multi-Objective Learning , 2006, The 2006 IEEE International Joint Conference on Neural Network Proceedings.

[77]  Leo Breiman,et al.  Classification and Regression Trees , 1984 .

[78]  Robert A. Jacobs,et al.  Increased rates of convergence through learning rate adaptation , 1987, Neural Networks.

[79]  James R. Williamson,et al.  Gaussian ARTMAP: A Neural Network for Fast Incremental Learning of Noisy Multidimensional Maps , 1996, Neural Networks.

[80]  Lester Ingber,et al.  Simulated annealing: Practice versus theory , 1993 .

[81]  Hisao Ishibuchi,et al.  Single-objective and two-objective genetic algorithms for selecting linguistic rules for pattern classification problems , 1997, Fuzzy Sets Syst..

[82]  Jong-Bae Park,et al.  An Improved Particle Swarm Optimization for , 2010 .

[83]  Gisbert Schneider,et al.  Optimized Particle Swarm Optimization (OPSO) and its application to artificial neural network training , 2006, BMC Bioinformatics.

[84]  Kiyong Choi,et al.  Parasitic-aware RF circuit design and optimization , 2004, IEEE Transactions on Circuits and Systems I: Regular Papers.

[85]  Erol Gelenbe,et al.  Traffic and video quality with adaptive neural compression , 1996, Multimedia Systems.

[86]  Hussein A. Abbass,et al.  Speeding Up Backpropagation Using Multiobjective Evolutionary Algorithms , 2003, Neural Computation.

[87]  Biing-Hwang Juang,et al.  Discriminative learning for minimum error classification [pattern recognition] , 1992, IEEE Trans. Signal Process..

[88]  Erol Gelenbe,et al.  Simulation with learning agents , 2001, Proc. IEEE.

[89]  J.R. Artalejo,et al.  G-networks: A versatile approach for work removal in queueing networks , 2000, Eur. J. Oper. Res..

[90]  Erol Gelenbe,et al.  Random Neural Networks with Negative and Positive Signals and Product Form Solution , 1989, Neural Computation.

[91]  Chia-Feng Juang,et al.  A hybrid of genetic algorithm and particle swarm optimization for recurrent network design , 2004, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[92]  Mansooreh Mollaghasemi,et al.  An Adaptive Multiobjective Approach to Evolving ART Architectures , 2010, IEEE Transactions on Neural Networks.

[93]  C. Hubert,et al.  Pattern completion with the random neural network using the RPROP learning algorithm , 1993, Proceedings of IEEE Systems Man and Cybernetics Conference - SMC.

[94]  Rainer Storn,et al.  Differential Evolution – A Simple and Efficient Heuristic for global Optimization over Continuous Spaces , 1997, J. Glob. Optim..

[95]  Erol Gelenbe,et al.  Random Neural Networks with Multiple Classes of Signals , 1999, Neural Computation.

[96]  Michael Georgiopoulos,et al.  Fuzzy ART properties , 1995, Neural Networks.

[97]  Georgios Loukas,et al.  A Biologically Inspired pired Denial of Service Detector Using the Random Neural Network , 2007, 2007 IEEE Internatonal Conference on Mobile Adhoc and Sensor Systems.

[98]  Gerardo Rubino,et al.  Performance evaluation of real-time speech through a packet network: a random neural networks-based approach , 2004, Perform. Evaluation.

[99]  Bernhard Sendhoff,et al.  Pareto-Based Multiobjective Machine Learning: An Overview and Case Studies , 2008, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[100]  Ju-Jang Lee,et al.  Training Two-Layered Feedforward Networks With Variable Projection Method , 2008, IEEE Transactions on Neural Networks.

[101]  Georgios C. Anagnostopoulos,et al.  Novel approaches in adaptive resonance theory for machine learning , 2001 .

[102]  Gerardo Rubino,et al.  A study of real-time packet video quality using random neural networks , 2002, IEEE Trans. Circuits Syst. Video Technol..

[103]  Erol Gelenbe,et al.  Stability of the Random Neural Network Model , 1990, Neural Computation.

[104]  Jonathan E. Fieldsend,et al.  Multi-class ROC analysis from a multi-objective optimisation perspective , 2006, Pattern Recognit. Lett..

[105]  Carlos A. Coello Coello,et al.  Evolutionary multi-objective optimization: a historical view of the field , 2006, IEEE Comput. Intell. Mag..

[106]  Lifeng Xi,et al.  An Improved Particle Swarm Optimization for Evolving Feedforward Artificial Neural Networks , 2007, Neural Processing Letters.

[107]  Stelios Timotheou.  Nonnegative Least Squares Learning for the Random Neural Network , 2008, ICANN.

[108]  Erol Gelenbe,et al.  Matched neural filters for EMI based mine detection , 1999, IJCNN'99. International Joint Conference on Neural Networks. Proceedings (Cat. No.99CH36339).

[109]  Moncef Gabbouj,et al.  Evolutionary artificial neural networks by multi-dimensional particle swarm optimization , 2009, Neural Networks.

[110]  Erol Gelenbe,et al.  Synchronized Interactions in Spiked Neuronal Networks , 2008, Comput. J..

[111]  Hans-Michael Voigt,et al.  Soft Genetic Operators in Evolutionary Algorithms , 1995, Evolution and Biocomputation.

[112]  Erol Gelenbe,et al.  Steps toward self-aware networks , 2009, CACM.

[113]  Teresa B. Ludermir,et al.  Particle Swarm Optimization of Neural Network Architectures and Weights , 2007 .

[114]  Stephen Grossberg,et al.  Fuzzy ARTMAP: A neural network architecture for incremental supervised learning of analog multidimensional maps , 1992, IEEE Trans. Neural Networks.

[115]  Arjen van Ooyen,et al.  Improving the convergence of the back-propagation algorithm , 1992, Neural Networks.

[116]  Chih-Jen Lin,et al.  LIBSVM: A library for support vector machines , 2011, TIST.

[117]  Juan Julián Merelo Guervós,et al.  Evolving Multilayer Perceptrons , 2000, Neural Processing Letters.

[118]  R. Eberhart,et al.  Fuzzy adaptive particle swarm optimization , 2001, Proceedings of the 2001 Congress on Evolutionary Computation (IEEE Cat. No.01TH8546).

[119]  Erol Gelenbe,et al.  Random Neural Networks with Synchronized Interactions , 2008, Neural Computation.

[120]  E. Gelenbe.  Product-form queueing networks with negative and positive customers , 1991 .

[121]  Michael Georgiopoulos,et al.  Order of Search in Fuzzy ART and Fuzzy ARTMAP: Effect of the Choice Parameter , 1996, Neural Networks.

[122]  Stelios Timotheou,et al.  The Random Neural Network: A Survey , 2010, Comput. J..

[123]  Yu Wang,et al.  Adaptive Inertia Weight Particle Swarm Optimization , 2006, ICAISC.

[124]  Xin Yao,et al.  Evolving artificial neural networks , 1999, Proc. IEEE.

[125]  Taskin Koçak,et al.  Design and implementation of a random neural network routing engine , 2003, IEEE Trans. Neural Networks.

[126]  Erol Gelenbe,et al.  The Random Neural Network Model for Texture Generation , 1992, Int. J. Pattern Recognit. Artif. Intell..

[127]  Erol Gelenbe,et al.  Design and performance of cognitive packet networks , 2001, Perform. Evaluation.

[128]  Antoniya Georgieva,et al.  Neural Network Learning With Global Heuristic Search , 2007, IEEE Transactions on Neural Networks.

[129]  Anastasios N. Venetsanopoulos,et al.  Fast learning algorithms for neural networks , 1992 .

[130]  Erol Gelenbe,et al.  Random Neural Network Recognition of Shaped Objects in Strong Clutter , 1997, ICANN.

[131]  Mohammad Bagher Menhaj,et al.  Training feedforward networks with the Marquardt algorithm , 1994, IEEE Trans. Neural Networks.

[132]  David E. Goldberg,et al.  Genetic Algorithms in Search Optimization and Machine Learning , 1988 .

[133]  Erol Gelenbe,et al.  Learning in the multiple class random neural network , 2002, IEEE Trans. Neural Networks.

[134]  Khaled F. Hussain,et al.  Laser intensity vehicle classification system based on random neural network , 2005, ACM-SE 43.

[135]  Laxmidhar Behera,et al.  On Adaptive Learning Rate That Guarantees Convergence in Feedforward Networks , 2006, IEEE Transactions on Neural Networks.