Sample re-weighting hyper box classifier for multi-class data classification

We propose two multi-class data classification methods.A sample weighting scheme has been incorporated into a literature classifier.Benchmark datasets demonstrate efficiency of the proposed classifier.A data partition method is proposed that reduce computational cost. In this work, we propose two novel classifiers for multi-class classification problems using mathematical programming optimisation techniques. A hyper box-based classifier (Xu & Papageorgiou, 2009) that iteratively constructs hyper boxes to enclose samples of different classes has been adopted. We firstly propose a new solution procedure that updates the sample weights during each iteration, which tweaks the model to favour those difficult samples in the next iteration and therefore achieves a better final solution. Through a number of real world data classification problems, we demonstrate that the proposed refined classifier results in consistently good classification performance, outperforming the original hyper box classifier and a number of other state-of-the-art classifiers.Furthermore, we introduce a simple data space partition method to reduce the computational cost of the proposed sample re-weighting hyper box classifier. The partition method partitions the original dataset into two disjoint regions, followed by training sample re-weighting hyper box classifier for each region respectively. Through some real world datasets, we demonstrate the data space partition method considerably reduces the computational cost while maintaining the level of prediction accuracies.

[1]  William Stafford Noble,et al.  Support vector machine , 2013 .

[2]  K. Johana,et al.  Benchmarking Least Squares Support Vector Machine Classifiers , 2022 .

[3]  Tzu-Tsung Wong,et al.  A hybrid discretization method for naïve Bayesian classifiers , 2012, Pattern Recognit..

[4]  Alfredo Ferro,et al.  Microarray Data Analysis: From Preparation To Classification , 2013 .

[5]  Lazaros G. Papageorgiou,et al.  AN MILP MODEL FOR MULTI-CLASS DATA CLASSIFICATION , 2006 .

[6]  Chee Peng Lim,et al.  A hybrid intelligent system for medical data classification , 2014, Expert Syst. Appl..

[7]  Toshiyuki Sueyoshi,et al.  DEA-discriminant analysis in the view of goal programming , 1999, Eur. J. Oper. Res..

[8]  Willy Gochet,et al.  Mathematical programming based heuristics for improving LP-generated classifiers for the multiclass supervised classification problem , 2006, Eur. J. Oper. Res..

[9]  Lazaros G. Papageorgiou,et al.  A mixed integer optimisation model for data classification , 2009, Comput. Ind. Eng..

[10]  Hao Yu,et al.  Selection of Proper Neural Network Sizes and Architectures—A Comparative Study , 2012, IEEE Transactions on Industrial Informatics.

[11]  Yoav Freund,et al.  A decision-theoretic generalization of on-line learning and an application to boosting , 1995, EuroCOLT.

[12]  Leo Breiman,et al.  Bagging Predictors , 1996, Machine Learning.

[13]  Metin Turkay,et al.  Prediction of folding type of proteins using mixed-integer linear programming , 2005 .

[14]  Emilio Carrizosa,et al.  Supervised classification and mathematical optimization , 2013, Comput. Oper. Res..

[15]  Wei Zhang,et al.  An Improvement to Naive Bayes for Text Classification , 2011 .

[16]  Michael R. Bussieck,et al.  General Algebraic Modeling System (GAMS) , 2004 .

[17]  Petros Xanthopoulos,et al.  A weighted support vector machine method for control chart pattern recognition , 2014, Comput. Ind. Eng..

[18]  E. Dougherty,et al.  Accurate and Reliable Cancer Classification Based on Probabilistic Inference of Pathway Activity , 2009, PloS one.

[19]  Luis Enrique Sucar,et al.  Learning an Optimal Naive Bayes Classifier , 2006, 18th International Conference on Pattern Recognition (ICPR'06).

[20]  Beatrice Gralton,et al.  Washington DC - USA , 2008 .

[21]  G. Tutz,et al.  An introduction to recursive partitioning: rationale, application, and characteristics of classification and regression trees, bagging, and random forests. , 2009, Psychological methods.

[22]  Mark H. Karwan,et al.  Combining a new data classification technique and regression analysis to predict the Cost-To-Serve new customers , 2011, Comput. Ind. Eng..

[23]  J. Downing,et al.  Classification, subtype discovery, and prediction of outcome in pediatric acute lymphoblastic leukemia by gene expression profiling. , 2002, Cancer cell.

[24]  Thomas J. Watson,et al.  An empirical study of the naive Bayes classifier , 2001 .

[25]  Hong Seo Ryoo,et al.  Pattern classification by concurrently determined piecewise linear and convex discriminant functions , 2006, Comput. Ind. Eng..

[26]  N. Kumarappan,et al.  A neural network approach to day-ahead deregulated electricity market prices classification , 2012 .

[27]  John J. Glen,et al.  An iterative mixed integer programming method for classification accuracy maximizing discriminant analysis , 2003, Comput. Oper. Res..

[28]  J. Hardin Microarray Data from a , 2005 .

[29]  Hasan Bal,et al.  A new mathematical programming approach to multi-group classification problems , 2011, Comput. Oper. Res..

[30]  Jason Weston,et al.  Gene Selection for Cancer Classification using Support Vector Machines , 2002, Machine Learning.

[31]  Wen Li,et al.  Two-level hierarchical combination method for text classification , 2011, Expert Syst. Appl..

[32]  Alaleh Maskooki,et al.  Improving the efficiency of a mixed integer linear programming based approach for multi-class classification problem , 2013, Comput. Ind. Eng..

[33]  Adil M. Bagirov,et al.  An efficient algorithm for the incremental construction of a piecewise linear classifier , 2011, Inf. Syst..

[34]  Toshiyuki Sueyoshi,et al.  Mixed integer programming approach of extended DEA-discriminant analysis , 2004, Eur. J. Oper. Res..

[35]  Hakan A. Çirpan,et al.  A set of new Chebyshev kernel functions for support vector machine pattern classification , 2011, Pattern Recognit..

[36]  Hasan Bal,et al.  An experimental comparison of the new goal programming and the linear programming approaches in the two-group discriminant problems , 2006, Comput. Ind. Eng..

[37]  Jurandy Almeida,et al.  Spam filtering: how the dimensionality reduction affects the accuracy of Naive Bayes classifiers , 2011, Journal of Internet Services and Applications.

[38]  I. Halil Kavakli,et al.  Classification of drug molecules considering their IC50 values using mixed-integer linear programming based hyper-boxes method , 2008, BMC Bioinformatics.

[39]  Jesús Angulo,et al.  Classification of hyperspectral images by tensor modeling and additive morphological decomposition , 2013, Pattern Recognit..

[40]  Jon Atli Benediktsson,et al.  Unsupervised methods for the classification of hyperspectral images with low spatial resolution , 2013, Pattern Recognit..

[41]  Ian H. Witten,et al.  The WEKA data mining software: an update , 2009, SKDD.

[42]  Toshiyuki Sueyoshi,et al.  DEA-DA for bankruptcy-based performance assessment: Misclassification analysis of Japanese construction industry , 2009, Eur. J. Oper. Res..

[43]  Taskin Kavzoglu,et al.  Increasing the accuracy of neural network classification using refined training data , 2009, Environ. Model. Softw..

[44]  R. Spang,et al.  Predicting the clinical status of human breast cancer by using gene expression profiles , 2001, Proceedings of the National Academy of Sciences of the United States of America.

[45]  Jingjing Lu,et al.  Comparing naive Bayes, decision trees, and SVM with AUC and accuracy , 2003, Third IEEE International Conference on Data Mining.

[46]  Young-Chan Lee,et al.  Bankruptcy prediction using support vector machine with optimal choice of kernel function parameters , 2005, Expert Syst. Appl..

[47]  Toby Walsh,et al.  Empirical Methods in AI , 1998, AI Mag..

[48]  Guangzhe Fan,et al.  Classification tree analysis using TARGET , 2008, Comput. Stat. Data Anal..

[49]  Metin Turkay,et al.  A mixed-integer programming approach to multi-class data classification problem , 2006, Eur. J. Oper. Res..

[50]  Laura Diosan,et al.  Improving classification performance of Support Vector Machine by genetically optimising kernel shape and hyper-parameters , 2010, Applied Intelligence.

[51]  Irina Rish,et al.  An empirical study of the naive Bayes classifier , 2001 .

[52]  Lior Rokach,et al.  Taxonomy for characterizing ensemble methods in classification tasks: A review and annotated bibliography , 2009, Comput. Stat. Data Anal..

[53]  Sebastian Nowozin,et al.  gBoost: a mathematical programming approach to graph classification and regression , 2009, Machine Learning.

[54]  H. Chipman,et al.  Bayesian CART Model Search , 1998 .

[55]  Benjamin Friedlander,et al.  An efficient algorithm , 1983 .

[56]  W. Gehrlein General mathematical programming formulations for the statistical classification problem , 1986 .

[57]  Wang Zhen,et al.  Non-parallel planes support vector machine for multi-class classification , 2010, 2010 International Conference on Logistics Systems and Intelligent Management (ICLSIM).

[58]  Banu Soylu,et al.  Multi-criteria inventory classification with reference items , 2014, Comput. Ind. Eng..

[59]  Kemal Polat,et al.  Classification of epileptiform EEG using a hybrid system based on decision tree classifier and fast Fourier transform , 2007, Appl. Math. Comput..

[60]  Li-Ching Ma,et al.  A two-phase case-based distance approach for multiple-group classification problems , 2012, Comput. Ind. Eng..

[61]  Toshiyuki Sueyoshi,et al.  DEA-Discriminant Analysis: Methodological comparison among eight discriminant analysis approaches , 2006, Eur. J. Oper. Res..

[62]  Wencong Lu,et al.  Predicting toxic action mechanisms of phenols using AdaBoost Learner , 2009 .

[63]  Kyuseok Shim,et al.  PUBLIC: A Decision Tree Classifier that Integrates Building and Pruning , 1998, Data Mining and Knowledge Discovery.

[64]  Toshiyuki Sueyoshi,et al.  Extended DEA-Discriminant Analysis , 2001, Eur. J. Oper. Res..

[65]  Si Wu,et al.  Improving support vector machine classifiers by modifying kernel functions , 1999, Neural Networks.

[66]  Danh V. Nguyen,et al.  Tumor classification by partial least squares using microarray gene expression data , 2002, Bioinform..

[67]  David G. Lowe,et al.  Local Naive Bayes Nearest Neighbor for image classification , 2011, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[68]  Kim Fung Lam,et al.  Combining discriminant methods in solving classification problems in two-group discriminant analysis , 2002, Eur. J. Oper. Res..

[69]  Aixia Guo,et al.  Gene Selection for Cancer Classification using Support Vector Machines , 2014 .

[70]  Abdesselam Bouzerdoum,et al.  A generalized feedforward neural network architecture for classification and regression , 2003, Neural Networks.

[71]  Doheon Lee,et al.  Inferring Pathway Activity toward Precise Disease Classification , 2008, PLoS Comput. Biol..

[72]  Lorenzo Bruzzone,et al.  Classification of hyperspectral remote-sensing data with primal SVM for small-sized training dataset problem☆ , 2008 .

[73]  I. Halil Kavakli,et al.  Optimization Based Tumor Classification from Microarray Gene Expression Data , 2011, PloS one.

[74]  Yi Lu Murphey,et al.  Multiclass pattern classification using neural networks , 2004, ICPR 2004.

[75]  Minghe Sun,et al.  A Mathematical Programming Approach for Gene Selection and Tissue Classification , 2003, Bioinform..

[76]  Shyam Visweswaran,et al.  The application of naive Bayes model averaging to predict Alzheimer's disease from genome-wide data , 2011, J. Am. Medical Informatics Assoc..

[77]  Dimitris Bertsimas,et al.  Classification and Regression via Integer Optimization , 2007, Oper. Res..

[78]  Joakim Nivre AN EFFICIENT ALGORITHM , 2003 .

[79]  Fabrizio Sebastiani,et al.  Machine learning in automated text categorization , 2001, CSUR.

[80]  Yi Lu Murphey,et al.  Multi-class pattern classification using neural networks , 2007, Pattern Recognit..

[81]  Eric Bauer,et al.  An Empirical Comparison of Voting Classification Algorithms: Bagging, Boosting, and Variants , 1999, Machine Learning.

[82]  Yong Shi,et al.  Several multi-criteria programming methods for classification , 2009, Comput. Oper. Res..