Meta-learning from experiment databases : an illustration

Gaining insights into the behavior of learning algorithms generally involves studying the performance of the algorithms on many different datasets, with various parameter settings, and analyzing the results. A useful method to learn effectively from previous learning episodes are experiment databases: databases designed to store detailed descriptions of a large number of experiments, selected to cover a wide range of conditions. We illustrate how such databases allow us to easily gain insights into a wide range of questions on algorithm behavior by just querying them and interpreting the results, or by using data mining methods to automatically discover patterns in learning algorithm performance. By putting these databases online, they serve as a repository of experimental results that can be (re)used by various researchers to easily obtain new insights.

[1]  Ian H. Witten,et al.  Data mining - practical machine learning tools and techniques, Second Edition , 2005, The Morgan Kaufmann series in data management systems.

[2]  Jeffrey S. Simonoff,et al.  Tree Induction Vs Logistic Regression: A Learning Curve Analysis , 2001, J. Mach. Learn. Res..

[3]  Robert C. Holte,et al.  Very Simple Classification Rules Perform Well on Most Commonly Used Datasets , 1993, Machine Learning.

[4]  Huan Liu,et al.  Book review: Machine Learning, Neural and Statistical Classification Edited by D. Michie, D.J. Spiegelhalter and C.C. Taylor (Ellis Horwood Limited, 1994) , 1996, SGAR.

[5]  M. Hilario,et al.  Building algorithm profiles for prior model selection in knowledge discovery systems , 1999, IEEE SMC'99 Conference Proceedings. 1999 IEEE International Conference on Systems, Man, and Cybernetics (Cat. No.99CH37028).

[6]  Hendrik Blockeel,et al.  Experiment Databases , 2007, Inductive Databases and Constraint-Based Data Mining.

[7]  Maarten van Someren.  Model Class Selection and Construction: Beyond the Procrustean Approach to Machine Learning Applications , 2001, Machine Learning and Its Applications.

[8]  David J. Spiegelhalter,et al.  Machine Learning, Neural and Statistical Classification , 2009 .

[9]  Peter A. Flach,et al.  Improved Dataset Characterisation for Meta-learning , 2002, Discovery Science.

[10]  Hendrik Blockeel.  Experiment Databases: A Novel Methodology for Experimental Research , 2005, KDID.

[11]  Walter Daelemans,et al.  Comparing Learning Approaches to Coreference Resolution. There is More to it Than 'Bias' , 2005, ICML 2005.