Multinomial logistic regression and product unit neural network models: Application of a new hybrid methodology for solving a classification problem in the livestock sector

This work presents a new approach for multi-class pattern recognition based on the hybridization of a linear and nonlinear model. We propose multinomial logistic regression where some new covariates are defined by a product unit neural network, where in turn, the nonlinear basis functions are constructed with the product of the inputs raised to arbitrary powers. The application of this methodology involves, first of all, training the coefficients and the basis structure of product unit models using techniques based on artificial neural networks and evolutionary algorithms, followed by the application of multinomial logistic regression to both the new derived features and the original ones. To evaluate the efficacy of our technique we pose a difficult problem, the classification of sheep with respect to their milk production in different lactations, using covariates that only involve the first weeks of lactation. This enables the productive capacity of the animal to be identified more rapidly and leads to a faster selection process in determining the best producers. The results obtained with our approach are compared to other classification methodologies. Although several of these methodologies offer good results, the percentage of cases correctly classified was higher with our approach, which shows how instrumental the potential use of this methodology is for decision making in livestock enterprises, a sector relatively untouched by the technological innovations in business management that have been appearing in the last few years.

[1]  Shigeo Abe DrEng Pattern Classification , 2001, Springer London.

[2]  Ryszard S. Michalski,et al.  Machine learning: an artificial intelligence approach volume III , 1990 .

[3]  Sergio Javier Pérez Olivero,et al.  Factores ambientales que influyen en la producción lechera en la raza ovina Manchega , 1994 .

[4]  David E. Goldberg,et al.  Genetic Algorithms and Walsh Functions: Part II, Deception and Its Analysis , 1989, Complex Syst..

[5]  John R. Anderson,et al.  MACHINE LEARNING An Artificial Intelligence Approach , 2009 .

[6]  Cheng-Jian Lin,et al.  A novel hybrid learning algorithm for parametric fuzzy CMAC networks and its classification applications , 2008, Expert Syst. Appl..

[7]  J. Ross Quinlan,et al.  C4.5: Programs for Machine Learning , 1992 .

[8]  Yen-Jen Oyang,et al.  Data classification with radial basis function networks based on a novel kernel density estimation algorithm , 2005, IEEE Transactions on Neural Networks.

[9]  Andries Petrus Engelbrecht,et al.  Global optimization algorithms for training product unit neural networks , 2000, Proceedings of the IEEE-INNS-ENNS International Joint Conference on Neural Networks. IJCNN 2000. Neural Computing: New Challenges and Perspectives for the New Millennium.

[10]  Ian H. Witten,et al.  Weka machine learning algorithms in java , 2000 .

[11]  María Pilar Molina Pons Composicion y factores de variacion de la leche de ovejas de raza manchega , 1988 .

[12]  Ali A. Minai,et al.  Back-propagation heuristics: a study of the extended delta-bar-delta algorithm , 1990, 1990 IJCNN International Joint Conference on Neural Networks.

[13]  César Hervás-Martínez,et al.  Evolutionary product unit based neural networks for regression , 2006, Neural Networks.

[14]  Cliff T. Ragsdale,et al.  Aggregating expert predictions in a networked environment , 2001, Comput. Oper. Res..

[15]  Eric R. Ziegel,et al.  Generalized Linear Models , 2002, Technometrics.

[16]  Vasant Honavar,et al.  Evolutionary Design of Neural Architectures -- A Preliminary Taxonomy and Guide to Literature , 1995 .

[17]  David G. Stork,et al.  Pattern Classification , 1973 .

[18]  Heekuck Oh,et al.  Neural Networks for Pattern Recognition , 1993, Adv. Comput..

[19]  F. Glover,et al.  Simple but powerful goal programming models for discriminant problems , 1981 .

[20]  María Pilar Molina Pons,et al.  Producción de leche: factores de variación , 1994 .

[21]  Parag C. Pendharkar,et al.  A threshold-varying artificial neural network approach for classification and its application to bankruptcy prediction problem , 2005, Comput. Oper. Res..

[22]  J. Mark Introduction to radial basis function networks , 1996 .

[23]  D. Hosmer,et al.  Applied Logistic Regression , 1991 .

[24]  Leo Breiman,et al.  Technical note: Some properties of splitting criteria , 2004, Machine Learning.

[25]  J. Friedman,et al.  Projection Pursuit Regression , 1981 .

[26]  Michael Schmitt,et al.  On the Complexity of Computing and Learning with Multiplicative Neural Networks , 2002, Neural Computation.

[27]  Ester Bernadó-Mansilla,et al.  Accuracy-Based Learning Classifier Systems: Models, Analysis and Applications to Classification Tasks , 2003, Evolutionary Computation.

[28]  P. McCullagh,et al.  Generalized Linear Models , 1992 .

[29]  Roy Rada,et al.  Expert systems and evolutionary computing for financial investing: A review , 2008, Expert Syst. Appl..

[30]  Sunghoon Kwon,et al.  Multiclass sparse logistic regression for classification of multiple cancer types using gene expression data , 2006, Comput. Stat. Data Anal..

[31]  W. Vent,et al.  Rechenberg, Ingo, Evolutionsstrategie — Optimierung technischer Systeme nach Prinzipien der biologischen Evolution. 170 S. mit 36 Abb. Frommann‐Holzboog‐Verlag. Stuttgart 1973. Broschiert , 1975 .

[32]  Frederick Kaefer,et al.  A neural network application to consumer classification to improve the timing of direct marketing activities , 2005, Comput. Oper. Res..

[33]  David E. Goldberg,et al.  Genetic Algorithms and Walsh Functions: Part I, A Gentle Introduction , 1989, Complex Syst..

[34]  T. Minka A comparison of numerical optimizers for logistic regression , 2004 .

[35]  J. Ross Quinlan,et al.  Induction of Decision Trees , 1986, Machine Learning.

[36]  Abdulhamit Subasi,et al.  Wavelet neural network classification of EEG signals by using AR model with MLE preprocessing , 2005, Neural Networks.

[37]  Panos M. Pardalos,et al.  MULTICRITERIA SORTING METHODOLOGY: APPLICATION TO FINANCIAL DECISION PROBLEMS , 2000, Parallel Algorithms Appl..

[38]  Tian-Shyug Lee,et al.  Mining the customer credit using classification and regression tree and multivariate adaptive regression splines , 2006, Comput. Stat. Data Anal..

[39]  D. Cox,et al.  The analysis of binary data , 1971 .

[40]  Michael Y. Hu,et al.  Two-Group Classification Using Neural Networks* , 1993 .

[41]  J. Freidman,et al.  Multivariate adaptive regression splines , 1991 .

[42]  D. Cox,et al.  Analysis of Binary Data (2nd ed.). , 1990 .

[43]  A. C. Martínez-Estudillo,et al.  Hybridization of evolutionary algorithms and local search by means of a clustering method , 2005, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[44]  César Hervás-Martínez,et al.  Logistic regression using covariates obtained by product-unit neural network models , 2007, Pattern Recognit..

[45]  David E. Rumelhart,et al.  Product Units: A Computationally Powerful and Biologically Plausible Extension to Backpropagation Networks , 1989, Neural Computation.

[46]  Yuan-chin Ivan Chang,et al.  Synergy of Logistic Regression and Support Vector Machine in Multiple-Class Classification , 2004, IDEAL.

[47]  Bernard Widrow,et al.  Layered neural nets for pattern recognition , 1988, IEEE Trans. Acoust. Speech Signal Process..

[48]  Ingo Rechenberg,et al.  Evolutionsstrategie : Optimierung technischer Systeme nach Prinzipien der biologischen Evolution , 1973 .

[49]  César Hervás-Martínez,et al.  Approximating the sheep milk production curve through the use of artificial neural networks and genetic algorithms , 2005, Comput. Oper. Res..

[50]  R. Tibshirani,et al.  Generalized Additive Models , 1991 .

[51]  David E. Goldberg,et al.  Genetic Algorithms in Search Optimization and Machine Learning , 1988 .

[52]  C. J. Stone,et al.  Polychotomous Regression , 1995 .

[53]  John A. Nelder,et al.  Generalized linear models. 2nd ed. , 1993 .

[54]  Lucila Ohno-Machado,et al.  Logistic regression and artificial neural network classification models: a methodology review , 2002, J. Biomed. Informatics.

[55]  Maha N. Hajmeer,et al.  Comparison of logistic regression and neural network-based classifiers for bacterial growth , 2003 .

[56]  Eric R. Ziegel,et al.  Analysis of Binary Data (2nd ed.) , 1991 .

[57]  Carlos Buxadé Carbó Ovino de leche: aspectos claves , 1997 .

[58]  W. Vach,et al.  On the misuses of artificial neural networks for prognostic and diagnostic classification in oncology. , 2000, Statistics in medicine.

[59]  Randall S Sexton,et al.  Employee turnover: a neural network solution , 2005, Comput. Oper. Res..

[60]  Michael Georgiopoulos,et al.  Coupling weight elimination with genetic algorithms to reduce network size and preserve generalization , 1997, Neurocomputing.

[61]  Stanley Lemeshow,et al.  Applied Logistic Regression, Second Edition , 1989 .

[62]  Paz Lavín,et al.  Producción de leche de oveja: la región de Castilla y León (España) como modelo , 2000 .

[63]  Vasant Honavar,et al.  Evolutionary Design of Neural Architectures , 1995 .

[64]  Richard A. Johnson,et al.  Applied Multivariate Statistical Analysis , 1983 .