A hybrid ensemble pruning approach based on consensus clustering and multi-objective evolutionary algorithm for sentiment classification

Sentiment analysis is a critical task of extracting subjective information from online text documents. Ensemble learning can be employed to obtain more robust classification schemes. However, most approaches in the field incorporated feature engineering to build efficient sentiment classifiers.The purpose of our research is to establish an effective sentiment classification scheme by pursuing the paradigm of ensemble pruning. Ensemble pruning is a crucial method to build classifier ensembles with high predictive accuracy and efficiency. Previous studies employed exponential search, randomized search, sequential search, ranking based pruning and clustering based pruning. However, there are tradeoffs in selecting the ensemble pruning methods. In this regard, hybrid ensemble pruning schemes can be more promising.In this study, we propose a hybrid ensemble pruning scheme based on clustering and randomized search for text sentiment classification. Furthermore, a consensus clustering scheme is presented to deal with the instability of clustering results. The classifiers of the ensemble are initially clustered into groups according to their predictive characteristics. Then, two classifiers from each cluster are selected as candidate classifiers based on their pairwise diversity. The search space of candidate classifiers is explored by the elitist Pareto-based multi-objective evolutionary algorithm.For the evaluation task, the proposed scheme is tested on twelve balanced and unbalanced benchmark text classification tasks. In addition, the proposed approach is experimentally compared with three ensemble methods (AdaBoost, Bagging and Random Subspace) and three ensemble pruning algorithms (ensemble selection from libraries of models, Bagging ensemble selection and LibD3C algorithm). Results demonstrate that the consensus clustering and the elitist pareto-based multi-objective evolutionary algorithm can be effectively used in ensemble pruning. The experimental analysis with conventional ensemble methods and pruning algorithms indicates the validity and effectiveness of the proposed scheme.

[1]  Anil K. Jain.  Data clustering: 50 years beyond K-means , 2010, Pattern Recognit. Lett..

[2]  Andries P. Engelbrecht,et al.  Computational Intelligence: An Introduction , 2002 .

[3]  Rich Caruana,et al.  Ensemble selection from libraries of models , 2004, ICML.

[4]  Alípio Mário Jorge,et al.  Ensemble approaches for regression: A survey , 2012, CSUR.

[5]  Hamido Fujita,et al.  A hybrid approach to the sentiment analysis problem at the sentence level , 2016, Knowl. Based Syst..

[6]  Aytug Onan,et al.  A feature selection model based on genetic rank aggregation for text sentiment classification , 2017, J. Inf. Sci..

[7]  Thomas G. Dietterich,et al.  Pruning Adaptive Boosting , 1997, ICML.

[8]  Yu Wang,et al.  Ensemble classification based on supervised clustering for credit scoring , 2016, Appl. Soft Comput..

[9]  Gang Wang,et al.  Mining affective text to improve social media item recommendation , 2015, Inf. Process. Manag..

[10]  El-Ghazali Talbi,et al.  Metaheuristics - From Design to Implementation , 2009 .

[11]  Ian H. Witten,et al.  The WEKA data mining software: an update , 2009, SKDD.

[12]  Mahdi Eftekhari,et al.  A new ensemble learning methodology based on hybridization of classifier ensemble selection approaches , 2015, Appl. Soft Comput..

[13]  Aytug Onan,et al.  Ensemble of keyword extraction methods and classifiers in text classification , 2016, Expert Syst. Appl..

[14]  Basilio Sierra,et al.  Classifier Subset Selection to construct multi-classifiers by means of estimation of distribution algorithms , 2015, Neurocomputing.

[15]  Teuvo Kohonen,et al.  Self-Organizing Maps , 2010 .

[16]  Haytham Elghazel,et al.  Ensemble multi-label text categorization based on rotation forest and latent semantic indexing , 2016, Expert Syst. Appl..

[17]  S. N. Sivanandam,et al.  Introduction to genetic algorithms , 2007 .

[18]  Zhu Zhang,et al.  POS-RS: A Random Subspace method for sentiment classification based on part-of-speech analysis , 2015, Inf. Process. Manag..

[19]  Fernando Jiménez,et al.  Multi-objective evolutionary algorithms for fuzzy classification in survival prediction , 2014, Artif. Intell. Medicine.

[20]  Sandro Vega-Pons,et al.  A Survey of Clustering Ensemble Algorithms , 2011, Int. J. Pattern Recognit. Artif. Intell..

[21]  Rui Xia,et al.  Ensemble of feature sets and classification algorithms for sentiment classification , 2011, Inf. Sci..

[22]  Wei Tang,et al.  Ensembling neural networks: Many could be better than all , 2002, Artif. Intell..

[23]  Qun Dai,et al.  ModEnPBT: A Modified Backtracking Ensemble Pruning algorithm , 2013, Appl. Soft Comput..

[24]  Huaxiang Zhang,et al.  A spectral clustering based ensemble pruning approach , 2014, Neurocomputing.

[25]  Lior Rokach,et al.  Ensemble-based classifiers , 2010, Artificial Intelligence Review.

[26]  Ludmila I. Kuncheva,et al.  Measures of Diversity in Classifier Ensembles and Their Relationship with the Ensemble Accuracy , 2003, Machine Learning.

[27]  Subhash C. Bagui,et al.  Combining Pattern Classifiers: Methods and Algorithms , 2005, Technometrics.

[28]  Jiawei Han,et al.  Data Mining: Concepts and Techniques , 2000 .

[29]  Matti Aksela,et al.  Comparison of Classifier Selection Methods for Improving Committee Performance , 2003, Multiple Classifier Systems.

[30]  Thiago J. M. Moura,et al.  Combining diversity measures for ensemble pruning , 2016, Pattern Recognit. Lett..

[31]  Larry S. Yaeger,et al.  Building a General Purpose Cross-Domain Sentiment Mining Model , 2009, 2009 WRI World Congress on Computer Science and Information Engineering.

[32]  Boris G. Mirkin,et al.  Reinterpreting the Category Utility Function , 2001, Machine Learning.

[33]  Dongmei Zhang,et al.  An ensemble method for unbalanced sentiment classification , 2015, 2015 11th International Conference on Natural Computation (ICNC).

[34]  Stanislaw Osowski,et al.  Aggregation of classifiers ensemble using local discriminatory power and quantiles , 2016, Expert Syst. Appl..

[35]  Gonzalo Martínez-Muñoz,et al.  Pruning in ordered bagging ensembles , 2006, ICML.

[36]  Xin Jin,et al.  Expectation Maximization Clustering , 2010, Encyclopedia of Machine Learning.

[37]  Bogdan Gabrys,et al.  Application of the Evolutionary Algorithms for Classifier Selection in Multiple Classifier Systems with Majority Voting , 2001, Multiple Classifier Systems.

[38]  Qun Dai,et al.  A novel ensemble pruning algorithm based on randomized greedy selective strategy and ballot , 2013, Neurocomputing.

[39]  Gonzalo Martínez-Muñoz,et al.  Using boosting to prune bagging ensembles , 2007, Pattern Recognit. Lett..

[40]  Ting Zhang,et al.  A new reverse reduce-error ensemble pruning algorithm , 2015, Appl. Soft Comput..

[41]  Elisabetta Fersini,et al.  Expressive signals in social media languages to improve polarity detection , 2016, Inf. Process. Manag..

[42]  Fernando José Von Zuben,et al.  The Influence of the Pool of Candidates on the Performance of Selection and Combination Techniques in Ensembles , 2006, The 2006 IEEE International Joint Conference on Neural Network Proceedings.

[43]  Grigorios Tsoumakas,et al.  Ensemble Pruning Using Reinforcement Learning , 2006, SETN.

[44]  Chen Lin,et al.  LibD3C: Ensemble classifiers with a clustering and dynamic selection strategy , 2014, Neurocomputing.

[45]  Rainer Storn,et al.  Differential Evolution – A Simple and Efficient Heuristic for global Optimization over Continuous Spaces , 1997, J. Glob. Optim..

[46]  Joydeep Ghosh,et al.  Cluster ensembles , 2011, Data Clustering: Algorithms and Applications.

[47]  Enrico Glaab,et al.  Analysing functional genomics data using novel ensemble, consensus and data fusion techniques , 2011 .

[48]  Yong Qi,et al.  Information Processing and Management , 1984 .

[49]  Harith Alani,et al.  Contextual semantics for sentiment analysis of Twitter , 2016, Inf. Process. Manag..

[50]  Thomas G. Dietterich.  Multiple Classifier Systems , 2000, Lecture Notes in Computer Science.

[51]  Zhi Liu,et al.  Sentiment recognition of online course reviews using multi-swarm optimization-based selected features , 2016, Neurocomputing.

[52]  Sergios Theodoridis,et al.  Pattern Recognition, Third Edition , 2006 .

[53]  Mark A. Hall,et al.  Correlation-based Feature Selection for Machine Learning , 2003 .

[54]  Fabio Roli,et al.  Methods for Designing Multiple Classifier Systems , 2001, Multiple Classifier Systems.

[55]  Zhiqiang Cao,et al.  Sampled-data based average consensus of second-order integral multi-agent systems: Switching topologies and communication noises , 2013, Autom..

[56]  John H. Holland,et al.  Adaptation in Natural and Artificial Systems: An Introductory Analysis with Applications to Biology, Control, and Artificial Intelligence , 1992 .

[57]  C. Mallows,et al.  A Method for Comparing Two Hierarchical Clusterings , 1983 .

[58]  Ana L. N. Fred,et al.  Combining multiple clusterings using evidence accumulation , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[59]  Francisco Herrera,et al.  Ordering-based pruning for improving the performance of ensembles of classifiers in the framework of imbalanced datasets , 2016, Inf. Sci..

[60]  Paolo Rosso,et al.  Detecting positive and negative deceptive opinions using PU-learning , 2015, Inf. Process. Manag..

[61]  Quan Sun,et al.  Bagging Ensemble Selection , 2011, Australasian Conference on Artificial Intelligence.

[62]  Leo Breiman,et al.  Bagging Predictors , 1996, Machine Learning.

[63]  Akshi Kumar,et al.  Information Retrieval and Machine Learning: Supporting Technologies for Web Mining Research and Practice , 2008, Webology.

[64]  Estevam R. Hruschka,et al.  Tweet sentiment analysis with classifier ensembles , 2014, Decis. Support Syst..

[65]  XuFeng,et al.  Polarity shift detection, elimination and ensemble , 2016 .

[66]  Daniel Hernández-Lobato,et al.  Empirical analysis and evaluation of approximate techniques for pruning regression bagging ensembles , 2011, Neurocomputing.

[67]  Rudy Prabowo,et al.  Sentiment analysis: A combined approach , 2009, J. Informetrics.

[68]  Min Song,et al.  Opinion polarity detection in Twitter data combining shrinkage regression and topic modeling , 2016, J. Informetrics.

[69]  Vipin Kumar,et al.  Introduction to Data Mining, (First Edition) , 2005 .

[70]  Vipin Kumar,et al.  Introduction to Data Mining , 2022, Data Mining and Machine Learning Applications.

[71]  Wei Tang,et al.  Selective Ensemble of Decision Trees , 2003, RSFDGrC.

[72]  Elisabetta Fersini,et al.  Sentiment analysis: Bayesian Ensemble Learning , 2014, Decis. Support Syst..

[73]  Amit P. Sheth,et al.  Extracting Diverse Sentiment Expressions with Target-Dependent Polarity from Twitter , 2012, ICWSM.

[74]  Joydeep Ghosh,et al.  Cluster Ensembles --- A Knowledge Reuse Framework for Combining Multiple Partitions , 2002, J. Mach. Learn. Res..

[75]  R. Anitha,et al.  Malware detection by pruning of parallel ensembles using harmony search , 2013, Pattern Recognit. Lett..

[76]  Eibe Frank,et al.  Large-scale attribute selection using wrappers , 2009, 2009 IEEE Symposium on Computational Intelligence and Data Mining.

[77]  Qun Dai,et al.  Several novel evaluation measures for rank-based ensemble pruning with applications to time series prediction , 2015, Expert Syst. Appl..

[78]  Tony R. Martinez,et al.  Decision Tree Ensemble: Small Heterogeneous Is Better Than Large Homogeneous , 2008, 2008 Seventh International Conference on Machine Learning and Applications.

[79]  Nathalie Aussenac-Gilles,et al.  A study on LIWC categories for opinion mining in Spanish reviews , 2014, J. Inf. Sci..

[80]  Shokri Z. Selim,et al.  K-Means-Type Algorithms: A Generalized Convergence Theorem and Characterization of Local Optimality , 1984, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[81]  D. Rubin,et al.  Maximum likelihood from incomplete data via the EM - algorithm plus discussions on the paper , 1977 .

[82]  Hamidah Ibrahim,et al.  A Survey: Clustering Ensembles Techniques , 2009 .

[83]  Usman Qamar,et al.  TOM: Twitter opinion mining framework using hybrid classification scheme , 2014, Decis. Support Syst..

[84]  Jian Ma,et al.  Sentiment classification: The contribution of ensemble learning , 2014, Decis. Support Syst..

[85]  James Kennedy,et al.  Particle swarm optimization , 2002, Proceedings of ICNN'95 - International Conference on Neural Networks.