CTMS : A Comparative Text Mining System

In many applications, there is often a need for comparing multiple text collections to find commonalities and differences in topical themes, a task we refer to as comparative text mining . In this paper, we present a general comparative mining system (CTMS). The CTMS system takes any two collections of text and generates a list of cross-collection themes and their associated individual collection-specific themes. The themes are linked to representative passages in each collection. The themes are represented as word distributions, and the underlying comparative mining algorithm is based on a probabilistic mixture model. The system carries out all the stages of text mining from data cleaning and preprocessing to the actual mining and post-processing, allowing users to perform comparative analysis between any two collections and navigate through the extracted theme space. This system can potentially be applied to a broad range of areas including opinion summarization, business intelligence, and summarization of text.