The Implementation of Crawling News Page Based on Incremental Web Crawler

Web crawler technology is the technology which downloads web pages through the program. This paper implements incremental Python web crawler, uses Scrapy crawler framework, crawls news web pages from mainstream web sites incrementally in real time, and deposits data in the database. The key technology of incremental crawling is removing the repetition of web links, and the most common method is using Bloom filter. This paper implements a simplified Bloom filter. The result shows that the web crawler can monitor the news pages well.