Cached Suucient Statistics for Eecient Machine Learning with Large Datasets 1. Caching Suucient Statistics

This paper introduces new algorithms and data structures for quick counting for machine learning datasets. We focus on the counting task of constructing contingency tables, but our approach is also applicable to counting the number of records in a dataset that match conjunctive queries. Subject to certain assumptions, the costs of these operations can be shown to be independent of the number of records in the dataset and loglinear in the number of non-zero entries in the contingency table. We provide a very sparse data structure, the ADtree, to minimize memory use. We provide analytical worst-case bounds for this structure for several models of data distribution. We empirically demonstrate that tractably-sized data structures can be produced for large real-world datasets by (a) using a sparse tree structure that never allocates memory for counts of zero, (b) never allocating memory for counts that can be deduced from other counts, and (c) not bothering to expand the tree fully near its leaves. We show how the ADtree can be used to accelerate Bayes net structure nding algorithms, rule learning algorithms, and feature selection algorithms, and we provide a number of empirical results comparing ADtree methods against traditional direct counting approaches. We also discuss the possible uses of ADtrees in other machine learning methods, and discuss the merits of ADtrees in comparison with alternative representations such as kd-trees, R-trees and Frequent Sets. 1. Caching Su cient Statistics Computational e ciency is an important concern for machine learning algorithms, especially when applied to large datasets (Fayyad, Mannila, & Piatetsky-Shapiro, 1997; Fayyad & Uthurusamy, 1996) or in real-time scenarios. In earlier work we showed how kd-trees with multiresolution cached regression matrix statistics can enable very fast locally weighted and instance based regression (Moore, Schneider, & Deng, 1997). In this paper, we attempt to accelerate predictions for symbolic attributes using a kind of kd-tree that splits on all dimensions at all nodes. Many machine learning algorithms operating on datasets of symbolic attributes need to do frequent counting. This work is also applicable to Online Analytical Processing (OLAP) applications in data mining, where operations on large datasets such as multidimensional database access, DataCube operations (Harinarayan, Rajaraman, & Ullman, 1996), and association rule learning (Agrawal, Mannila, Srikant, Toivonen, & Verkamo, 1996) could be accelerated by fast counting. Let us begin by establishing some notation. We are given a data set with R records and M attributes. The attributes are called a1; a2; : : :aM . The value of attribute ai in the c 1998 AI Access Foundation and Morgan Kaufmann Publishers. All rights reserved.