Knowledge Discovery, Data Mining and Hybrid Systems

In the last two decades digitial revolution has invaded business enterprises. Computers have enabled organizations to store gigabytes of data related to stock markets, electricity consumption profiles, troubleshooting and diagnostic data, etc.. As outlined by Fayyad and Uthuruswamy (1996a), in scientific endeavors, data represents observations carefully collected about some phenomenon under study. In business, data captures information about critical markets, competitors, and customers. On the other hand, in manufacturing, data captures performance and optimization opportunities, as well as the keys to improving process and troubleshooting problems. The reason organizations store or collect all this data is to enable them to extract some useful knowledge (at a later date!!) which can make them more productive, efficient, and competitive. The terms, Knowledge Discovery in Databases (KDD), and Data Mining have emerged in the last five years from this need of extracting useful knowledge. KDD is a nontrivial process of identifying valid, novel, potentially useful, and ultimately understandable knowledge from data (Fayyad et al. 1996b). Data mining is a step in the KDD process that consists of applying data analysis and discovery (learning) algorithms that produce a particular enumeration of patterns (or models) over the data. In fact, the data mining process can fit into the framework of intelligent transformation systems described in chapter 3. It will be noticed in this chapter that the transformation process is largely a bottom up knowledge engineering strategy.