Optimal constraint-based decision tree induction from itemset lattices

In this article we show that there is a strong connection between decision tree learning and local pattern mining. This connection allows us to solve the computationally hard problem of finding optimal decision trees in a wide range of applications by post-processing a set of patterns: we use local patterns to construct a global model. We exploit the connection between constraints in pattern mining and constraints in decision tree induction to develop a framework for categorizing decision tree mining constraints. This framework allows us to determine which model constraints can be pushed deeply into the pattern mining process, and allows us to improve the state-of-the-art of optimal decision tree induction.

[1]  Bernhard Ganter,et al.  Formal Concept Analysis: Mathematical Foundations , 1998 .

[2]  Petko Valtchev,et al.  Yet a Faster Algorithm for Building the Hasse Diagram of a Concept Lattice , 2009, ICFCA.

[3]  Siegfried Nijssen,et al.  Pattern-Based Classification: A Unifying Perspective , 2011, ArXiv.

[4]  Arno J. Knobbe,et al.  Maximally informative k-itemsets and their efficient discovery , 2006, KDD '06.

[5]  Daniel Kifer,et al.  DualMiner: A Dual-Pruning Algorithm for Itemsets with Constraints , 2002, Data Mining and Knowledge Discovery.

[6]  Art Lew,et al.  Optimal conversion of extended-entry decision tables with general cost criteria , 1978, CACM.

[7]  Hiroki Arimura,et al.  LCM ver. 2: Efficient Mining Algorithms for Frequent/Closed/Maximal Itemsets , 2004, FIMI.

[8]  Jean-François Boulicaut,et al.  Approximation of Frequency Queris by Means of Free-Sets , 2000, PKDD.

[9]  H. Chipman,et al.  Bayesian CART Model Search , 1998 .

[10]  Jian Pei,et al.  CMAR: accurate and efficient classification based on multiple class-association rules , 2001, Proceedings 2001 IEEE International Conference on Data Mining.

[11]  Andrew W. Moore,et al.  Cached Sufficient Statistics for Efficient Machine Learning with Large Datasets , 1998, J. Artif. Intell. Res..

[12]  J. Ross Quinlan,et al.  C4.5: Programs for Machine Learning , 1992 .

[13]  William S. Meisel,et al.  An Algorithm for Constructing Optimal Binary Decision Trees , 1977, IEEE Transactions on Computers.

[14]  Wynne Hsu,et al.  Integrating Classification and Association Rule Mining , 1998, KDD.

[15]  Nicolas Pasquier,et al.  Efficient Mining of Association Rules Using Closed Itemset Lattices , 1999, Inf. Syst..

[16]  Laks V. S. Lakshmanan,et al.  Mining frequent itemsets with convertible constraints , 2001, Proceedings 17th International Conference on Data Engineering.

[17]  Michael J. Pazzani,et al.  Exploring the Decision Forest , 1993 .

[18]  Heikki Mannila,et al.  A database perspective on knowledge discovery , 1996, CACM.

[19]  Ashwin Machanavajjhala,et al.  l-Diversity: Privacy Beyond k-Anonymity , 2006, ICDE.

[20]  Johannes Fürnkranz,et al.  From Local Patterns to Global Models: The LeGo Approach to Data Mining , 2008 .

[21]  Pierangela Samarati,et al.  Protecting Respondents' Identities in Microdata Release , 2001, IEEE Trans. Knowl. Data Eng..

[22]  Russell Greiner,et al.  Exploring the Decision Forest: An Empirical Investigation of Occam's Razor in Decision Tree Induction , 1997 .

[23]  Ian H. Witten,et al.  Data mining: practical machine learning tools and techniques, 3rd Edition , 1999 .

[24]  Jean-François Boulicaut,et al.  Free-Sets: A Condensed Representation of Boolean Data for the Approximation of Frequency Queries , 2004, Data Mining and Knowledge Discovery.

[25]  A. Asuncion,et al.  UCI Machine Learning Repository, University of California, Irvine, School of Information and Computer Sciences , 2007 .

[26]  James Cussens,et al.  Exploiting Informative Priors for Bayesian Classification and Regression Trees , 2005, IJCAI.

[27]  T. Imielinski,et al.  A database perspective on knowledge discovery : A database perspective on knowledge discovery , 1996 .

[28]  Ramakrishnan Srikant,et al.  Fast Algorithms for Mining Association Rules in Large Databases , 1994, VLDB.

[29]  Jiawei Han,et al.  Summarizing itemset patterns: a profile-based approach , 2005, KDD '05.

[30]  Wray L. Buntine,et al.  Learning classification trees , 1992 .

[31]  Shaul Markovitch,et al.  Anytime Induction of Cost-sensitive Trees , 2007, NIPS.

[32]  Bart Goethals,et al.  Memory issues in frequent itemset mining , 2004, SAC '04.

[33]  Yoshua Bengio,et al.  Inference for the Generalization Error , 1999, Machine Learning.

[34]  Kyuseok Shim,et al.  Building Decision Trees with Constraints , 2001 .

[35]  Michael J. Pazzani,et al.  Exploring the Decision Forest: An Empirical Investigation of Occam's Razor in Decision Tree Induction , 1993, J. Artif. Intell. Res..

[36]  Srinivasan Parthasarathy,et al.  Parallel Algorithms for Discovery of Association Rules , 1997, Data Mining and Knowledge Discovery.

[37]  Latanya Sweeney,et al.  k-Anonymity: A Model for Protecting Privacy , 2002, Int. J. Uncertain. Fuzziness Knowl. Based Syst..

[38]  M. Garey.  Optimal Binary Identification Procedures , 1972 .

[39]  Peter D. Turney.  Cost-Sensitive Classification: Empirical Evaluation of a Hybrid Genetic Decision Tree Induction Algorithm , 1994, J. Artif. Intell. Res..

[40]  Shaul Markovitch,et al.  Anytime Learning of Decision Trees , 2007, J. Mach. Learn. Res..

[41]  Wei-Yin Loh,et al.  Classification and regression trees , 2011, WIREs Data Mining Knowl. Discov..

[42]  William S. Meisel,et al.  A Partitioning Algorithm with Application in Pattern Classification and the Optimization of Decision Trees , 1973, IEEE Transactions on Computers.

[43]  Heikki Mannila,et al.  Fast Discovery of Association Rules , 1996, Advances in Knowledge Discovery and Data Mining.

[44]  Kenneth C. Sevcik,et al.  The synthetic approach to decision table conversion , 1976, CACM.

[45]  Klaus-Robert Müller,et al.  Optimal dyadic decision trees , 2007, Machine Learning.

[46]  Ran Wolff,et al.  k-Anonymous Decision Tree Induction , 2006, PKDD.

[47]  Jian Pei,et al.  Mining frequent patterns without candidate generation , 2000, SIGMOD '00.

[48]  F. Bonchi,et al.  Extending the state-of-the-art of constraint-based pattern discovery , 2007, Data Knowl. Eng..

[49]  Jinyan Li,et al.  CAEP: Classification by Aggregating Emerging Patterns , 1999, Discovery Science.

[50]  Srinivasan Parthasarathy,et al.  New Algorithms for Fast Discovery of Association Rules , 1997, KDD.

[51]  Ian Witten,et al.  Data Mining , 2000 .

[52]  Ronald L. Rivest,et al.  Constructing Optimal Binary Decision Trees is NP-Complete , 1976, Inf. Process. Lett..

[53]  Luc De Raedt,et al.  Constraint-Based Pattern Set Mining , 2007, SDM.

[54]  Heikki Mannila,et al.  Efficient Algorithms for Discovering Association Rules , 1994, KDD Workshop.

[55]  Bart Goethals,et al.  Advances in frequent itemset mining implementations: report on FIMI'03 , 2004, SKDD.