Ranking of Classifiers based on Dataset Characteristics using Active Meta Learning

Classification is a machine learning technique which is used to categorize the different input patterns into different classes. To select the best classifier for a given dataset is one of the critical issues in Classification. Using cross-validation approach, it is possible to apply candidate algorithms on a given dataset and best classifier is selected by considering various evaluation measures of Classification. But computational cost is significant. Meta Learning automates this process by acquiring knowledge in form of Meta-features and performance information of candidate algorithm on datasets and creates a Meta Knowledge Base. Once Meta Knowledge Base is generated, system uses k-Nearest Neighbor as a Meta Learner that identifies the most similar datasets to new dataset. But generation of Meta Example is a costly process due to a large number of candidate algorithms and datasets with different characteristics involved. So Active Learning is incorporated into Meta Learning System that reduces generation of Meta example and at the same time maintaining performance of candidate algorithms. Once the training phase is completed based on Active Meta Learning approach, ranking is provided based on Success Rate Ratio (SRR) method that considers accuracy as a performance evaluation measure.

[1]  Mykola Pechenizkiy.  Data Mining Strategy Selection via Empirical and Constructive Induction , 2005, Databases and Applications.

[2]  Alexander Schliep,et al.  Ranking and selecting clustering algorithms using a meta-learning approach , 2008, 2008 IEEE International Joint Conference on Neural Networks (IEEE World Congress on Computational Intelligence).

[3]  Teresa Bernarda Ludermir,et al.  Uncertainty sampling methods for selecting datasets in active meta-learning , 2011, The 2011 International Joint Conference on Neural Networks.

[4]  Paul Davidsson,et al.  Analysis of Multi-Criteria Methods for Classifier and Algorithm Evaluation , 2007 .

[5]  Dilek Z. Hakkani-Tür,et al.  Active learning: theory and applications to automatic speech recognition , 2005, IEEE Transactions on Speech and Audio Processing.

[6]  Ricardo Vilalta,et al.  Introduction to the Special Issue on Meta-Learning , 2004, Machine Learning.

[7]  Teresa Bernarda Ludermir,et al.  Combining Uncertainty Sampling Methods for Active Meta-Learning , 2009, 2009 Ninth International Conference on Intelligent Systems Design and Applications.

[8]  Dunja Mladenic,et al.  kNN Versus SVM in the Collaborative Filtering Framework , 2006, Data Science and Classification.

[9]  Iain Paterson,et al.  The Focused Multi-Criteria Ranking Approach to Machine Learning Algorithm Selection-An Incremental Meta Learning Assistant for Data Mining Tasks , 2001 .

[10]  Myra Spiliopoulou,et al.  NOEMON: An Intelligent Assistant for Classifier Selection , 2007 .

[11]  Jun won Lee.  Relationships Among Learning Algorithms and Tasks , 2011 .

[12]  David A. Cohn,et al.  Improving generalization with active learning , 1994, Machine Learning.

[13]  Teresa Bernarda Ludermir,et al.  Selective generation of training examples in active meta-learning , 2008, Int. J. Hybrid Intell. Syst..

[14]  Teresa Bernarda Ludermir,et al.  Active Meta-Learning with Uncertainty Sampling and Outlier Detection , 2008, 2008 IEEE International Joint Conference on Neural Networks (IEEE World Congress on Computational Intelligence).

[15]  Carlos Soares,et al.  Zoomed Ranking: Selection of Classification Algorithms Based on Relevant Performance Information , 2000, PKDD.

[16]  Muhammad Zubair Shafiq,et al.  Guidelines to Select Machine Learning Scheme for Classification of Biomedical Datasets , 2009, EvoBIO.

[17]  Carlos Soares,et al.  Ranking Learning Algorithms: Using IBL and Meta-Learning on Accuracy and Time Results , 2003, Machine Learning.

[18]  Alexandros Kalousis,et al.  Algorithm selection via meta-learning , 2002 .

[19]  Ricardo Vilalta,et al.  Using Meta-Learning to Support Data Mining , 2004, Int. J. Comput. Sci. Appl..

[20]  Yulan He,et al.  An empirical framework for automatically selecting the best Bayesian classifier , 2009 .

[21]  Rodica Potolea,et al.  Evolutional meta-learning framework for automatic classifier selection , 2009, 2009 IEEE 5th International Conference on Intelligent Computer Communication and Processing.

[22]  Kate Smith-Miles,et al.  On learning algorithm selection for classification , 2006, Appl. Soft Comput..

[23]  Ion Muslea,et al.  Active Learning with Multiple Views , 2009, Encyclopedia of Data Warehousing and Mining.

[24]  Dana Angluin,et al.  Queries and concept learning , 1988, Machine Learning.