The Lumberjack Algorithm for Learning Linked Decision Forests

While the decision tree is an effective representation that has been used in many domains, a tree can often encode a concept inefficiently. This happens when the tree has to represent a subconcept multiple times in different parts of the tree. In this paper we introduce a new representation based on trees, the linked decision forest, that does not need to repeat internal structure. We also introduce the Lumberjack algorithm for growing these forests in a supervised learning setting. Lumberjack induces new subconcepts from repeated internal structure. This allows Lumberjack to represent many concepts more efficiently than a normal tree structure. We then show empirically that Lumberjack improves generalization accuracy on these hierarchically decomposable concepts.

[1]  Craig G. Nevill-Manning,et al.  Inferring Sequential Structure , 1996 .

[2]  J. Ross Quinlan,et al.  C4.5: Programs for Machine Learning , 1992 .

[3]  James A. Storer,et al.  Data compression via textual substitution , 1982, JACM.

[4]  Jesse Hoey,et al.  SPUDD: Stochastic Planning using Decision Diagrams , 1999, UAI.

[5]  Manuela M. Veloso,et al.  Tree Based Discretization for Continuous State Space Reinforcement Learning , 1998, AAAI/IAAI.

[6]  Manuela M. Veloso,et al.  The Lumberjack Algorithm for Learning Linked Decision Forests , 2000, PRICAI.

[7]  Alberto Maria Segre,et al.  Programs for Machine Learning , 1994 .

[8]  Leslie Pack Kaelbling,et al.  Input Generalization in Delayed Reinforcement Learning: An Algorithm and Performance Comparisons , 1991, IJCAI.

[9]  Randal E. Bryant,et al.  Symbolic Boolean manipulation with ordered binary-decision diagrams , 1992, CSUR.

[10]  Andrew McCallum,et al.  Reinforcement learning with selective perception and hidden state , 1996 .

[11]  C. S. Wallace,et al.  An Information Measure for Classification , 1968, Comput. J..

[12]  Ronald L. Rivest,et al.  Inferring Decision Trees Using the Minimum Description Length Principle , 1989, Inf. Comput..

[13]  J. Rissanen.  A UNIVERSAL PRIOR FOR INTEGERS AND ESTIMATION BY MINIMUM DESCRIPTION LENGTH , 1983 .

[14]  Michael J. Pazzani,et al.  Exploring the Decision Forest: An Empirical Investigation of Occam's Razor in Decision Tree Induction , 1993, J. Artif. Intell. Res..

[15]  Ron Kohavi,et al.  Wrappers for performance enhancement and oblivious decision graphs , 1995 .

[16]  Ian H. Witten,et al.  Identifying Hierarchical Structure in Sequences: A linear-time algorithm , 1997, J. Artif. Intell. Res..