Evolutionary product unit based neural networks for regression

This paper presents a new method for regression based on the evolution of a type of feed-forward neural networks whose basis function units are products of the inputs raised to real number power. These nodes are usually called product units. The main advantage of product units is their capacity for implementing higher order functions. Nevertheless, the training of product unit based networks poses several problems, since local learning algorithms are not suitable for these networks due to the existence of many local minima on the error surface. Moreover, it is unclear how to establish the structure of the network since, hitherto, all learning methods described in the literature deal only with parameter adjustment. In this paper, we propose a model of evolution of product unit based networks to overcome these difficulties. The proposed model evolves both the weights and the structure of these networks by means of an evolutionary programming algorithm. The performance of the model is evaluated in five widely used benchmark functions and a hard real-world problem of microbial growth modeling. Our evolutionary model is compared to a multistart technique combined with a Levenberg-Marquardt algorithm and shows better overall performance in the benchmark functions as well as the real-world problem.

[1]  Andries P. Engelbrecht,et al.  Computational Intelligence: An Introduction , 2002 .

[2]  César Hervás-Martínez,et al.  Multi-objective cooperative coevolution of artificial neural networks (multi-objective cooperative networks) , 2002, Neural Networks.

[3]  David E. Rumelhart,et al.  Product Units: A Computationally Powerful and Biologically Plausible Extension to Backpropagation Networks , 1989, Neural Computation.

[4]  Xin Yao,et al.  A new evolutionary system for evolving artificial neural networks , 1997, IEEE Trans. Neural Networks.

[5]  Hannu Korkeala,et al.  Evaluation of Lactobacillus sake Contamination in Vacuum-Packaged Sliced Cooked Meat Products by Ribotyping. , 1996, Journal of food protection.

[6]  Jenq-Neng Hwang,et al.  Regression modeling in back-propagation and projection pursuit learning , 1994, IEEE Trans. Neural Networks.

[7]  C. Lee Giles,et al.  Learning with Product Units , 1994, NIPS.

[8]  Dirk Tomandl,et al.  A Modified General Regression Neural Network (MGRNN) with new, efficient training algorithms as a robust 'black box'-tool for data analysis , 2001, Neural Networks.

[9]  César Hervás-Martínez,et al.  Performance of response surface model for prediction of Leuconostoc mesenteroides growth parameters under different experimental conditions , 2006 .

[10]  Jerome H. Friedman.  Multivariate adaptive regression splines (with discussion) , 1991 .

[11]  Pia Mäkelä.  Lactic acid bacterial contamination at meat processing plants : with special reference to ropy slime producing lactic acid bacteria , 1993 .

[12]  Refik Soyer,et al.  Bayesian Methods for Nonlinear Classification and Regression , 2004, Technometrics.

[13]  John Fulcher,et al.  Computational Intelligence: An Introduction , 2008, Computational Intelligence: A Compendium.

[14]  Volker Roth,et al.  The generalized LASSO , 2004, IEEE Transactions on Neural Networks.

[15]  John G Phillips,et al.  Response Surface Model for Predicting the Effects of Temperature pH, Sodium Chloride Content, Sodium Nitrite Concentration and Atmosphere on the Growth of Listeria monocytogenes. , 1990, Journal of food protection.

[16]  Kurt Hornik,et al.  Multilayer feedforward networks are universal approximators , 1989, Neural Networks.

[17]  C. Geyer,et al.  Annealing Markov chain Monte Carlo with applications to ancestral inference , 1995 .

[18]  R. H. Myers,et al.  Response Surface Methodology: Process and Product Optimization Using Designed Experiments , 1995 .

[19]  Christopher G. Atkeson,et al.  Implementing projection pursuit learning , 1996, IEEE Trans. Neural Networks.

[20]  Michael Schmitt,et al.  On the Complexity of Computing and Learning with Multiplicative Neural Networks , 2002, Neural Computation.

[21]  Peter J. Angeline,et al.  An evolutionary algorithm that constructs recurrent neural networks , 1994, IEEE Trans. Neural Networks.

[22]  Abdesselam Bouzerdoum,et al.  A generalized feedforward neural network architecture for classification and regression , 2003, Neural Networks.

[23]  Vladimir N. Vapnik,et al.  The Nature of Statistical Learning Theory , 2000, Statistics for Engineering and Information Science.

[24]  R. H. J. M. Otten,et al.  The Annealing Algorithm , 1989 .

[25]  C. Hervás-Martínez,et al.  Modelling the growth of Leuconostoc mesenteroides by Artificial Neural Networks. , 2005, International journal of food microbiology.

[26]  Shie-Jue Lee,et al.  An ART-based construction of RBF networks , 2002, IEEE Trans. Neural Networks.

[27]  J Baranyi,et al.  A dynamic approach to predicting bacterial growth in food. , 1994, International journal of food microbiology.

[28]  D. Signorini,et al.  Neural networks , 1995, The Lancet.

[29]  Andries Petrus Engelbrecht,et al.  Global optimization algorithms for training product unit neural networks , 2000, Proceedings of the IEEE-INNS-ENNS International Joint Conference on Neural Networks. IJCNN 2000. Neural Computing: New Challenges and Perspectives for the New Millennium.

[30]  J. Friedman.  Multivariate adaptive regression splines , 1990 .

[31]  Abdesselam Bouzerdoum,et al.  A generalized feedforward neural network classifier , 2003, Proceedings of the International Joint Conference on Neural Networks, 2003..

[32]  Harris Drucker,et al.  Improving Regressors using Boosting Techniques , 1997, ICML.

[33]  Its'hak Dinstein,et al.  A comparative study of neural network based feature extraction paradigms , 1999, Pattern Recognit. Lett..

[34]  James F. Frenzel,et al.  Training product unit neural networks with genetic algorithms , 1993, IEEE Expert.

[35]  C. D. Gelatt,et al.  Optimization by Simulated Annealing , 1983, Science.

[36]  Andries P. Engelbrecht,et al.  Pruning product unit neural networks , 2002, Proceedings of the 2002 International Joint Conference on Neural Networks. IJCNN'02 (Cat. No.02CH37290).

[37]  Robert L. Buchanan,et al.  Response surface models for the growth kinetics of Escherichia coli O157:H7 , 1993 .

[38]  J. H. Veld,et al.  Microbial and biochemical spoilage of foods: an overview , 1996 .

[39]  Donald F. Specht,et al.  A general regression neural network , 1991, IEEE Trans. Neural Networks.

[40]  César Hervás-Martínez,et al.  COVNET: a cooperative coevolutionary model for evolving artificial neural networks , 2003, IEEE Trans. Neural Networks.

[41]  Vladimir Cherkassky,et al.  Comparison of adaptive methods for function estimation from samples , 1996, IEEE Trans. Neural Networks.

[42]  Bart Kosko,et al.  Neural networks and fuzzy systems: a dynamical systems approach to machine intelligence , 1991 .

[43]  J Baranyi,et al.  Mathematics of predictive food microbiology. , 1995, International journal of food microbiology.