Adaptive Information Filtering: Concepts and Algorithms

Adaptive information filtering is concerned with filtering information streams in dynamic (changing) environments. The changes may occur both on the transmission side — the nature of the streams can change — and on the reception side — the interests of the user (or group of users) can change. While information filtering and information retrieval have a lot in common, this dissertation’s primary concern is with the differences. The temporal nature of information filtering necessitates more flexible document representation methods than does information retrieval where all the occurring terms are known in advance. Also, information filtering typically maintains user interest profiles requiring a learning system capable of coping with dynamic environments in place of the static queries characteristic of information retrieval. The research described in this dissertation investigates the employment of two distinct machine learning approaches, namely evolutionary computation (evolutionary algorithms) and neural computation (neural networks), for the intelligent optimization of incremental classification of information streams. The document representation employed in this research is weighted n-gram frequency distributions. The weights associated with the n-grams are the attributes being optimized. The results indicate the feasibility of the machine learning approach described in the previous paragraph. Written documents as well as spoken documents were succesfully classified, within the constraints posed by adaptive information filtering. The scalability issue requires further investigation: the classification results dropped from above 95% correct for two topics to below 85% correct for ten topics, although the drop in classification results seemed to level off above eight topics.