From Focused Crawling to Expert Information: an Application Framework for Web Exploration and Portal Generation

Publisher Summary Focused crawling is a relatively new, promising approach to improving the recall of expert search on the Web. It typically starts from a user- or community specific tree of topics along with a few training documents for each tree node, and then crawls the Web with focus on these topics of interest. This process can efficiently build a theme-specific, hierarchical directory whose nodes are populated with relevant high-quality documents for expert Web search. The BINGO! focused crawler implements an approach that aims to overcome the limitations of the initial training data. BINGO! identifies, among the crawled and positively classified documents of a topic, characteristic archetypes and uses them for periodically retraining the classifier. This way the crawler is dynamically adapted based on the most significant documents seen so far.