Constructive Neural Network Algorithms for Feedforward Architectures Suitable for Classification Tasks

This chapter presents and discusses several well-known constructive neural network algorithms suitable for constructing feedforward architectures aiming at classification tasks involving two classes. The algorithms are divided into two different groups: the ones directed by the minimization of classification errors and those based on a sequential model. In spite of the focus being on two-class classification algorithms, the chapter also briefly comments on the multiclass versions of several two-class algorithms, highlights some of the most popular constructive algorithms for regression problems and refers to several other alternative algorithms.

[1]  Nicolaos B. Karayiannis,et al.  Growing radial basis neural networks: merging supervised and unsupervised learning with network growth techniques , 1997, IEEE Trans. Neural Networks.

[2]  A. Wendemuth.  LEARNING THE UNLEARNABLE , 1995 .

[3]  Maria do Carmo Nicoletti,et al.  MBabCoNN - A Multiclass Version of a Constructive Neural Network Algorithm Based on Linear Separability and Convex Hull , 2008, ICANN.

[4]  Shang-Liang Chen,et al.  Orthogonal least squares learning algorithm for radial basis function networks , 1991, IEEE Trans. Neural Networks.

[5]  Colin Campbell,et al.  The Target Switch Algorithm: A Constructive Learning Procedure for Feed-Forward Neural Networks , 1995, Neural Computation.

[6]  Russell Reed,et al.  Pruning algorithms-a survey , 1993, IEEE Trans. Neural Networks.

[7]  Kagan Tumer,et al.  Structural adaptation and generalization in supervised feed-forward networks , 1994 .

[8]  Thomas R. Shultz,et al.  Could Knowledge-Based Neural Learning be Useful in Developmental Robotics? The Case of Kbcc , 2007, Int. J. Humanoid Robotics.

[9]  Stanley J. Farlow,et al.  Self-Organizing Methods in Modeling: Gmdh Type Algorithms , 1984 .

[10]  J. Friedman.  Exploratory Projection Pursuit , 1987 .

[11]  Mario Marchand,et al.  Learning by Minimizing Resources in Neural Networks , 1989, Complex Syst..

[12]  Marco Tomassini,et al.  Fuzzy Growing Hierarchical Self-Organizing Networks , 2008, ICANN.

[13]  Lutz Prechelt,et al.  Investigation of the CasCor Family of Learning Algorithms , 1997, Neural Networks.

[14]  John C. Platt.  A Resource-Allocating Network for Function Interpolation , 1991, Neural Computation.

[15]  Mario Marchand,et al.  On learning simple neural concepts: from halfspace intersections to neural decision lists , 1993 .

[16]  J.-P. Thivierge,et al.  Transferring domain rules in a constructive network: introducing RBCC , 2004, 2004 IEEE International Joint Conference on Neural Networks (IEEE Cat. No.04CH37541).

[17]  Shuyuan Yang,et al.  Incremental constructive ridgelet neural network , 2008, Neurocomputing.

[18]  Sheng Chen,et al.  Orthogonal least squares methods and their application to non-linear system identification , 1989 .

[19]  Albert Y. Zomaya,et al.  Toward generating neural network structures for function approximation , 1994, Neural Networks.

[20]  Jihoon Yang,et al.  MUpstart-a constructive neural network learning algorithm for multi-category pattern classification , 1997, Proceedings of International Conference on Neural Networks (ICNN'97).

[21]  Stephen I. Gallant,et al.  Perceptron-based learning algorithms , 1990, IEEE Trans. Neural Networks.

[22]  Tomas Hrycej,et al.  Modular Learning in Neural Networks: A Modularized Approach to Neural Network Classification , 1992 .

[23]  Edoardo Amaldi,et al.  Two Constructive Methods for Designing Compact Feedforward Networks of Threshold Units , 1997, Int. J. Neural Syst..

[24]  Yann LeCun,et al.  Optimal Brain Damage , 1989, NIPS.

[25]  Khashayar Khorasani,et al.  A new strategy for adaptively constructing multilayer feedforward neural networks , 2003, Neurocomputing.

[26]  Marcus R. Frean,et al.  A "Thermal" Perceptron Learning Rule , 1992, Neural Computation.

[27]  D. B. Fogel,et al.  Evolving neural networks , 1990, Biological Cybernetics.

[28]  F. Smieja.  Neural network constructive algorithms: Trading generalization for learning efficiency? , 1993 .

[29]  Juan Miguel Ortiz-de-Lazcano-Lobato,et al.  On the Generalization of the m-Class RDP Neural Network , 2008, ICANN.

[30]  Scott E. Fahlman,et al.  The Recurrent Cascade-Correlation Architecture , 1990, NIPS.

[31]  J. Nadal,et al.  Learning in feedforward layered networks: the tiling algorithm , 1989 .

[32]  Leonardo Franco,et al.  A New Decomposition Algorithm for Threshold Synthesis and Generalization of Boolean Functions , 2008, IEEE Transactions on Circuits and Systems I: Regular Papers.

[33]  Raquel L C Giordano,et al.  Online filtering of CO2 signals from a bioreactor gas outflow using a committee of constructive neural networks , 2008, Bioprocess and biosystems engineering.

[34]  Xin Yao,et al.  Evolving artificial neural networks , 1999, Proc. IEEE.

[35]  Leonardo Franco,et al.  Active Learning Using a Constructive Neural Network Algorithm , 2008, ICANN.

[36]  T. Shultz,et al.  Neural network modeling of developmental effects in discrimination shifts. , 1998, Journal of experimental child psychology.

[37]  Christian Lebiere,et al.  The Cascade-Correlation Learning Architecture , 1989, NIPS.

[38]  Timur Ash,et al.  Dynamic node creation in backpropagation networks , 1989 .

[39]  Christopher M. Bishop,et al.  Neural networks for pattern recognition , 1995 .

[40]  J. D. Schaffer,et al.  Combinations of genetic algorithms and neural networks: a survey of the state of the art , 1992, [Proceedings] COGANN-92: International Workshop on Combinations of Genetic Algorithms and Neural Networks.

[41]  W. Krauth,et al.  Learning algorithms with optimal stability in neural networks , 1987 .

[42]  Miroslav Kubat,et al.  Designing neural network architectures for pattern recognition , 2000, The Knowledge Engineering Review.

[43]  Thomas R. Shultz,et al.  Finding relevant knowledge: KBCC applied to DNA splice-junction determination , 2002, Proceedings of the 2002 International Joint Conference on Neural Networks. IJCNN'02 (Cat. No.02CH37290).

[44]  Leonardo Franco,et al.  Active Learning Using a Constructive Neural Network Algorithm , 2009, Constructive Neural Networks.

[45]  Marco Muselli,et al.  Switching Neural Networks: A New Connectionist Model for Classification , 2005, WIRN/NAIS.

[46]  Kazuyuki Murase,et al.  Self-Organization of Spiking Neural Network Generating Autonomous Behavior in a Real Mobile Robot , 2005, International Conference on Computational Intelligence for Modelling, Control and Automation and International Conference on Intelligent Agents, Web Technologies and Internet Commerce (CIMCA-IAWTIC'06).

[47]  J. Friedman,et al.  Projection Pursuit Regression , 1981 .

[48]  Maria do Carmo Nicoletti,et al.  An empirical evaluation of constructive neural network algorithms in classification tasks , 2007 .

[49]  Marcus Frean,et al.  The Upstart Algorithm: A Method for Constructing and Training Feedforward Neural Networks , 1990, Neural Computation.

[50]  Jihoon Yang,et al.  Pruning strategies for the MTiling constructive learning algorithm , 1997, Proceedings of International Conference on Neural Networks (ICNN'97).

[51]  Thomas R. Shultz,et al.  Development of Children's Seriation: A Connectionist Approach , 1999, Connect. Sci..

[52]  Nicolás García-Pedrajas,et al.  A cooperative constructive method for neural networks for pattern recognition , 2007, Pattern Recognit..

[53]  Tom Downs,et al.  CARVE-a constructive algorithm for real-valued examples , 1998, IEEE Trans. Neural Networks.

[54]  M. Golea,et al.  A Convergence Theorem for Sequential Learning in Two-Layer Perceptrons , 1990 .

[55]  Marco Muselli,et al.  Sequential constructive techniques , 1998 .

[56]  Jukka Saarinen,et al.  Estimating movements of a robotic manipulator by hybrid constructive neural networks , 2004, Neurocomputing.

[57]  Engelbert Mephu Nguifo,et al.  M-CLANN: Multi-class Concept Lattice-Based Artificial Neural Network for Supervised Classification , 2008, ICANN.

[58]  Thomas R. Shultz,et al.  A dual-phase technique for pruning constructive networks , 2003, Proceedings of the International Joint Conference on Neural Networks, 2003..

[59]  Sorin Draghici,et al.  The constraint based decomposition (CBD) training architecture , 2001, Neural Networks.

[60]  Eddy Mayoraz,et al.  Constructive Training Methods for feedforward Neural Networks with Binary weights , 1995, Int. J. Neural Syst..

[61]  Thomas R. Shultz,et al.  Application of knowledge-based cascade-correlation to vowel recognition , 2002, Proceedings of the 2002 International Joint Conference on Neural Networks. IJCNN'02 (Cat. No.02CH37290).

[62]  Geoffrey G. Towell,et al.  Symbolic knowledge and neural networks: insertion, refinement and extraction , 1992 .

[63]  Neil Burgess,et al.  A Constructive Algorithm that Converges for Real-Valued Input Patterns , 1994, Int. J. Neural Syst..

[64]  James T. Kwok,et al.  Objective functions for training new hidden units in constructive neural networks , 1997, IEEE Trans. Neural Networks.

[65]  Wlodzislaw Duch,et al.  Projection Pursuit Constructive Neural Networks Based on Quality of Projected Clusters , 2008, ICANN.

[66]  Michael Biehl,et al.  The AdaTron: An Adaptive Perceptron Algorithm , 1989 .

[67]  I C G Campbell,et al.  Constructive learning techniques for designing neural network systems , 1998 .

[68]  Mohamed Tajine,et al.  Adapting the 2-class recursive deterministic perceptron neural network to m-classes , 1997, Proceedings of International Conference on Neural Networks (ICNN'97).

[69]  E. Fiesler,et al.  Comparative Bibliography of Ontogenic Neural Networks , 1994 .

[70]  Thomas R. Shultz,et al.  Knowledge-based cascade-correlation , 2000, Proceedings of the IEEE-INNS-ENNS International Joint Conference on Neural Networks. IJCNN 2000. Neural Computing: New Challenges and Perspectives for the New Millennium.

[71]  Jude Shavlik,et al.  Refinement ofApproximate Domain Theories by Knowledge-Based Neural Networks , 1990, AAAI.

[72]  Cornelius T. Leondes,et al.  Neural network systems techniques and applications , 1998 .

[73]  Thomas R. Shultz,et al.  Knowledge-based cascade-correlation: Using knowledge to speed learning , 2001, Connect. Sci..

[74]  Mikko Lehtokangas.  Modelling with constructive backpropagation , 1999, Neural Networks.

[75]  Hervé Poulard,et al.  Training a neuron in sequential learning , 2004, Neural Processing Letters.

[76]  Stefan C. Kremer,et al.  Comments on "Constructive learning of recurrent neural networks: limitations of recurrent cascade correlation and a simple solution" , 1996, IEEE Trans. Neural Networks.

[77]  David A. Elizondo,et al.  A Reward-Value Based Constructive Method for the Autonomous Creation of Machine Controllers , 2008, ICANN.

[78]  Jukka Saarinen,et al.  Evaluation of constructive neural networks with cascaded architectures , 2002, Neurocomputing.

[79]  Marco Muselli,et al.  A Constructive Technique Based on Linear Programming for Training Switching Neural Networks , 2008, ICANN.

[80]  C. Lee Giles,et al.  Constructive learning of recurrent neural networks: limitations of recurrent cascade correlation and a simple solution , 1995, IEEE Trans. Neural Networks.

[81]  Khashayar Khorasani,et al.  New training strategies for constructive neural networks with application to regression problems , 2004, Neural Networks.

[82]  Jean-Pierre Nadal,et al.  Study of a Growth Algorithm for a Feedforward Network , 1989, Int. J. Neural Syst..

[83]  Di Wang,et al.  Fast Constructive-Covering Algorithm for neural networks and its implement in classification , 2008, Appl. Soft Comput..

[84]  Jihoon Yang,et al.  Constructive Neural-Network Learning Algorithms for Pattern Classification , 2000 .

[85]  James T. Kwok,et al.  Constructive algorithms for structure learning in feedforward neural networks for regression problems , 1997, IEEE Trans. Neural Networks.

[86]  Kazuyuki Murase,et al.  A new algorithm to design compact two-hidden-layer artificial neural networks , 2001, Neural Networks.

[87]  Fabio Massimo Frattale Mascioli,et al.  A constructive algorithm for binary neural networks: the oil-spot algorithm , 1995, IEEE Trans. Neural Networks.

[88]  Andreas Wichert,et al.  Pictorial reasoning with cell assemblies , 2001, Connect. Sci..