Improving Generalization of Neural Networks Using MLP Discriminant Based on Multiple Classifiers Failures

Multiple classifier systems or ensemble is an idea that is relevant both to neural computing and to machine learning community. Different MCSs can be designed for creating classifier ensembles with different combination functions. However, the best MCS can only be determined by performance evaluation. In this study, MCS is used to construct discriminant set that was used to discriminate the difficult to learn from the easy to learn patterns. Hence, this study explores several potentially productive ways in which an appropriate discriminant set or failure treatment might be developed based on the selection of the two failure cases: training failures and test failures. The experiments presented in this paper illustrate the application of discrimination techniques using multilayer perceptron (MLP) discriminants to neural networks trained to solve supervised learning task such as the Launch Interceptor Condition 1 problem. The experimental results reveal that directed splitting using an MLP discriminant is an important strategy in improving generalization of the networks.

[1]  Derek Partridge,et al.  Strategies for improving neural net generalisation , 2005, Neural Computing & Applications.

[2]  Thomas G. Dietterich.  Multiple Classifier Systems , 2000, Lecture Notes in Computer Science.

[3]  Giorgio Valentini,et al.  Ensembles of Learning Machines , 2002, WIRN.

[4]  Ethem Alpaydin,et al.  Multiple networks for function learning , 1993, IEEE International Conference on Neural Networks.

[5]  A. Taha,et al.  An experiment in software redundancy with diverse methodologies , 1992, Proceedings of the Twenty-Fifth Hawaii International Conference on System Sciences.

[6]  Thomas G. Dietterich.  Machine-Learning Research Four Current Directions , 1997 .

[7]  Milton S. Boyd,et al.  Designing a neural network for forecasting financial and economic time series , 1996, Neurocomputing.

[8]  Geoffrey E. Hinton,et al.  Learning internal representations by error propagation , 1986 .

[9]  Warren S. Sarle,et al.  Stopped Training and Other Remedies for Overfitting , 1995 .

[10]  Wei Lin,et al.  Multiple Classifiers Combination Based on Specialists' FIelds , 2006, 2006 Fifth Mexican International Conference on Artificial Intelligence.

[11]  Padraig Cunningham,et al.  Using Diversity in Preparing Ensembles of Classifiers Based on Different Feature Subsets to Minimize Generalization Error , 2001, ECML.

[12]  Bogdan Gabrys,et al.  Classifier selection for majority voting , 2005, Inf. Fusion.

[13]  Nancy G. Leveson,et al.  An experimental evaluation of the assumption of independence in multiversion programming , 1986, IEEE Transactions on Software Engineering.

[14]  Vasile Palade,et al.  Multi-Classifier Systems: Review and a roadmap for developers , 2006, Int. J. Hybrid Intell. Syst..

[15]  Fabio Roli,et al.  Multiple classifier systems : 4th International Workshop, MCS 2003, Guildford, UK, June 11-13, 2003 : proceedings , 2003 .

[16]  Josef Kittler,et al.  Multiple Classifier Systems , 2004, Lecture Notes in Computer Science.

[17]  Noel E. Sharkey,et al.  Neural computing for software reliability , 1994 .

[18]  Bev Littlewood,et al.  Conceptual Modeling of Coincident Failures in Multiversion Software , 1989, IEEE Trans. Software Eng..

[19]  Anders Krogh,et al.  Neural Network Ensembles, Cross Validation, and Active Learning , 1994, NIPS.

[20]  Tianzi Jiang,et al.  A combinational feature selection and ensemble neural network method for classification of gene expression data , 2004, BMC Bioinformatics.

[21]  Tin Kam Ho,et al.  Complexity of Classification Problems and Comparative Advantages of Combined Classifiers , 2000, Multiple Classifier Systems.

[22]  Ping Guo,et al.  Studies of model selection and regularization for generalization in neural networks with applications , 2002 .

[23]  Leo Breiman,et al.  Bagging Predictors , 1996, Machine Learning.

[24]  Ophir Frieder,et al.  IIT at TREC 2002 Linear Combinations Based on Document Structure and Varied Stemming for Arabic Retrieval , 2002, TREC.

[25]  Wei Tang,et al.  Selective Ensemble of Decision Trees , 2003, RSFDGrC.

[26]  D. Partridge.  Distinct Failure Diversity in Multiversion SoftwareDerek Partridge , 1997 .

[27]  Nikunj C. Oza,et al.  Decimated input ensembles for improved generalization , 1999, IJCNN'99. International Joint Conference on Neural Networks. Proceedings (Cat. No.99CH36339).

[28]  Fadzilah Siraj,et al.  WEBIC: a web based business insolvency classifier using neural networks , 2003 .

[29]  Hyeran Byun,et al.  Robust Face Detection and Tracking for Real-Life Applications , 2003, Int. J. Pattern Recognit. Artif. Intell..

[30]  C. J. Whitaker,et al.  Ten measures of diversity in classifier ensembles: limits for two classifiers , 2001 .

[31]  Josef Kittler,et al.  Multiple Classifiers Systems (MCS07) , 2007 .

[32]  Ramzi El-Haddadeh,et al.  Converged IP Network for Hydrocarbon Process Automation , 2010, 2010 Second International Conference on Computational Intelligence, Modelling and Simulation.

[33]  Abu Bakar,et al.  Design Of Normal Concrete Mixes Using Neural Network Model , 2000 .

[34]  William B. Yates,et al.  Engineering Multiversion Neural-Net Systems , 1996, Neural Computation.

[35]  Wei Tang,et al.  Ensembling neural networks: Many could be better than all , 2002, Artif. Intell..

[36]  Raymond J. Mooney,et al.  Constructing Diverse Classifier Ensembles using Artificial Training Examples , 2003, IJCAI.

[37]  Josef Kittler,et al.  Multiple classifier systems : second International Workshop, MCS 2001, Cambridge, UK, July 2-4, 2001 : proceedings , 2001 .