Discovery of frequent patterns in large data collections

Data mining or knowledge discovery in databases aims at nding useful reg ularities in large data sets Interest in the eld is motivated by the growth of computerized data collections and by the high potential value of patterns discovered in those collections For instance bar code readers at supermar kets produce extensive amounts of data about purchases An analysis of this data can reveal useful information about the shopping behavior of the customers Association rules for instance are a class of patterns that tell which products tend to be purchased together The general data mining task we consider is the following given a class of patterns that possibly have occurrences in a given data collection determine which patterns occur frequently and are thus probably the most useful ones It is characteristic for data mining applications to deal with high volumes of both data and patterns We address the algorithmic problems of determining e ciently which pat terns are frequent in the given data Our contributions are new algorithms analyses of problems and pattern classes for data mining We also present extensive experimental results We start by giving an e cient method for the discovery of all frequent association rules a well known data mining problem We then introduce the problem of discovering frequent patterns in general and show how the association rule algorithm can be extended to cover this problem We analyze the problem complexity and derive a lower bound for the number of queries in a simple but realistic model We then show how sampling can be used in the discovery of exact association rules and we give algorithms that are e cient especially in terms of the amount of database processing We also show that association rules with negation