Data Mining by Decomposition: Adaptive Search for Hypothesis Generation

Data mining methods search large databases for interesting patterns that may lead to useful decisions in organizations. When the database is defined over scores of attributes, the complexity of the search increases due to the combinatorial explosion at the attribute-space level, because billions of attribute subsets are candidates for forming interesting patterns in the database. A useful way to address this complexity is to partition the search problem and apply separate, but intertwined, algorithms for attribute search and pattern search. A genetic algorithm is applied on the attribute search problem to identify subsets that lead to more interesting patterns. This method is applied on a real world database arising from the investigations into the "Persian Gulf Illness." Computational experiments resulted in significant success compared to random or manual attribute selection.