Bayesian Learning of Neural Networks by Means of Artificial Immune Systems

Once the design of artificial neural networks (ANN) may require the optimization of numerical and structural parameters, bio-inspired algorithms have been successfully applied to accomplish this task, since they are population-based search strategies capable of dealing successfully with complex and large search spaces, avoiding local minima. In this paper, we propose the use of an artificial immune system for learning feedforward ANN's topologies. Besides the number of neurons in the hidden layer, the algorithm also optimizes the type of activation function for each node. The use of a Bayesian framework to infer the weights and weight decay terms as well as to perform model selection allows us to find neural models with high generalization capability and low complexity, once the Occam's razor principle is incorporated into the framework. We demonstrate the applicability of the proposal on seven classification problems and promising results were obtained.

[1]  Wray L. Buntine,et al.  Bayesian Back-Propagation , 1991, Complex Syst..

[2]  Jerne Nk.  Towards a network theory of the immune system. , 1974 .

[3]  Xin Yao,et al.  Evolving artificial neural networks , 1999, Proc. IEEE.

[4]  F T Vertosick,et al.  The immune system as a neural network: a multi-epitope approach. , 1991, Journal of theoretical biology.

[5]  Fernando José Von Zuben,et al.  Hybrid neural networks: An evolutionary approach with local search , 2002, Integr. Comput. Aided Eng..

[6]  Antonia J. Jones,et al.  Genetic algorithms and their applications to the design of neural networks , 1993, Neural Computing & Applications.

[7]  Catherine Blake,et al.  UCI Repository of machine learning databases , 1998 .

[8]  James T. Kwok,et al.  Constructive algorithms for structure learning in feedforward neural networks for regression problems , 1997, IEEE Trans. Neural Networks.

[9]  L.N. de Castro,et al.  An artificial immune network for multimodal function optimization , 2002, Proceedings of the 2002 Congress on Evolutionary Computation. CEC'02 (Cat. No.02TH8600).

[10]  Heekuck Oh,et al.  Neural Networks for Pattern Recognition , 1993, Adv. Comput..

[11]  David J. C. MacKay,et al.  A Practical Bayesian Framework for Backpropagation Networks , 1992, Neural Computation.

[12]  G. G. Stokes.  "J." , 1890, The New Yale Book of Quotations.

[13]  Byoung-Tak Zhang.  A Bayesian evolutionary approach to the design and learning of heterogeneous neural trees , 2002, Integr. Comput. Aided Eng..

[14]  D. Dasgupta.  Artificial Immune Systems and Their Applications , 1998, Springer Berlin Heidelberg.

[15]  Roberto Battiti,et al.  First- and Second-Order Methods for Learning: Between Steepest Descent and Newton's Method , 1992, Neural Computation.

[16]  M. Matteucci,et al.  EVOLUTIONARY LEARNING OF RICH NEURAL NETWORKS IN THE BAYESIAN MODEL SELECTION FRAMEWORK , 2004 .

[17]  David J. C. MacKay,et al.  Bayesian Interpolation , 1992, Neural Computation.

[18]  Russell Reed,et al.  Pruning algorithms-a survey , 1993, IEEE Trans. Neural Networks.

[19]  Fernando J. Von Zuben,et al.  Designing Ensembles of Fuzzy Classification Systems: An Immune-Inspired Approach , 2005, ICARIS.

[20]  Simon Haykin,et al.  Neural Networks: A Comprehensive Foundation , 1998 .

[21]  Xin Yao,et al.  Evolutionary design of artificial neural networks with different nodes , 1996, Proceedings of IEEE International Conference on Evolutionary Computation.

[22]  David J. C. MacKay,et al.  The Evidence Framework Applied to Classification Networks , 1992, Neural Computation.

[23]  Geoffrey E. Hinton,et al.  Bayesian Learning for Neural Networks , 1995 .

[24]  Leandro Nunes de Castro,et al.  An Immunological Approach to Initialize Feedforward Neural Network Weights , 2001 .

[25]  Christian Blum,et al.  Training feed-forward neural networks with ant colony optimization: an application to pattern classification , 2005, Fifth International Conference on Hybrid Intelligent Systems (HIS'05).

[26]  James T. Kwok,et al.  Bayesian Regularization in Constructive Neural Networks , 1996, ICANN.

[27]  G L Ada,et al.  The clonal-selection theory. , 1987, Scientific American.

[28]  DistAl: An inter-pattern distance-based constructive learning algorithm , 1999, Intell. Data Anal..

[29]  Hans Henrik Thodberg,et al.  A review of Bayesian neural networks with an application to near infrared spectroscopy , 1996, IEEE Trans. Neural Networks.

[30]  Peter J. Angeline,et al.  An evolutionary algorithm that constructs recurrent neural networks , 1994, IEEE Trans. Neural Networks.

[31]  Jonathan Timmis,et al.  Artificial immune systems - a new computational intelligence paradigm , 2002 .

[32]  Martin Fodslette Meiller.  A Scaled Conjugate Gradient Algorithm for Fast Supervised Learning , 1993 .

[33]  Fernando José Von Zuben,et al.  Copt-aiNet and the Gene Ordering Problem , 2003, WOB.

[34]  Fernando José Von Zuben,et al.  An immune-inspired approach to Bayesian networks , 2005, Fifth International Conference on Hybrid Intelligent Systems (HIS'05).