Anytime Induction of Decision Trees: An Iterative Improvement Approach

Most existing decision tree inducers are very fast due to their greedy approach. In many real-life applications, however, we are willing to allocate more time to get better decision trees. Our recently introduced LSID3 contract anytime algorithm allows computation speed to be traded for better tree quality. As a contract algorithm, LSID3 must be allocated its resources a priori, which is not always possible. In this work, we present IIDT, a general framework for interruptible induction of decision trees that need not be allocated resources a priori. The core of our proposed framework is an iterative improvement algorithm that repeatedly selects a subtree whose reconstruction is expected to yield the highest marginal utility. The algorithm then rebuilds the subtree with a higher allocation of resources. IIDT can also be configured to receive training examples as they become available, and is thus appropriate for incremental learning tasks. Empirical evaluation with several hard concepts shows that IIDT exhibits good anytime behavior and significantly outperforms greedy inducers when more time is available. A comparison of IIDT to several modern decision tree learners showed it to be superior.

[1]  J. Ross Quinlan,et al.  C4.5: Programs for Machine Learning , 1992 .

[2]  Pedro M. Domingos The Role of Occam's Razor in Knowledge Discovery , 1999, Data Mining and Knowledge Discovery.

[3]  Shaul Markovitch,et al.  Feature Generation Using General Constructor Functions , 2002, Machine Learning.

[4]  Leo Breiman,et al.  Bagging Predictors , 1996, Machine Learning.

[5]  Catherine Blake,et al.  UCI Repository of machine learning databases , 1998 .

[6]  Stuart J. Russell,et al.  Principles of Metareasoning , 1989, Artif. Intell..

[7]  Eric Joel Hovitz Computation and action under bounded resources , 1991 .

[8]  Ian Witten,et al.  Data Mining , 2000 .

[9]  Ronald L. Rivest,et al.  Constructing Optimal Binary Decision Trees is NP-Complete , 1976, Inf. Process. Lett..

[10]  Yoav Freund,et al.  The Alternating Decision Tree Learning Algorithm , 1999, ICML.

[11]  Vladimir N. Vapnik,et al.  The Nature of Statistical Learning Theory , 2000, Statistics for Engineering and Information Science.

[12]  Russell Greiner,et al.  Budgeted Learning of Naive-Bayes Classifiers , 2003, UAI.

[13]  Raymond J. Mooney,et al.  Symbolic and neural learning algorithms: An experimental comparison , 1991, Machine Learning.

[14]  Shaul Markovitch,et al.  Lookahead-based algorithms for anytime induction of decision trees , 2004, ICML.

[15]  Mark Craven,et al.  Extracting comprehensible models from trained neural networks , 1996 .

[16]  Paul E. Utgoff,et al.  Decision Tree Induction Based on Efficient Tree Restructuring , 1997, Machine Learning.

[17]  O. J. Murphy,et al.  Designing Storage Efficient Decision Trees , 1991, IEEE Trans. Computers.

[18]  Russell Greiner,et al.  Budgeted learning of nailve-bayes classifiers , 2002, UAI 2002.

[19]  Paul E. Utgoff,et al.  Incremental Induction of Decision Trees , 1989, Machine Learning.

[20]  Dimitrios Kalles,et al.  Breeding Decision Trees Using Evolutionary Techniques , 2001, ICML.

[21]  Shlomo Zilberstein,et al.  Optimal Composition of Real-Time Systems , 1996, Artif. Intell..

[22]  David W. Opitz,et al.  An anytime approach to connectionist theory refinement - refining the topologies of knowledge-based neural networks , 1996, Technical Report / University of Wisconsin, Madison / Computer Sciences Department.

[23]  David Haussler,et al.  Occam's Razor , 1987, Inf. Process. Lett..

[24]  Remco R. Bouckaert,et al.  Choosing Between Two Learning Algorithms Based on Calibrated Tests , 2003, ICML.

[25]  D. Ruppert The Elements of Statistical Learning: Data Mining, Inference, and Prediction , 2004 .

[26]  David Page,et al.  Sequential skewing: an improved skewing algorithm , 2004, ICML.