Extracting news text from web pages: an application for the visually impaired

Apart from the actual content, web pages contain several other components (referred to as boilerplate text) that describes how, and in what context the content should be displayed. We show how content bearing text can be efficiently separated from boilerplate text using a random forest classifier. We compare the performance with another state-of-the-art method for boilerplate detection that uses a decision tree classifier and shallow features extracted from the text. The result is a general improvement using the random forest classifier for both classifying problems analyzed, significantly so for the more complex problem. We also show that a small increase in feature set range can lead to even further improved accuracy. The conclusion is that random forest classification can achieve significantly higher accuracy rates than at least one of the current state-of-the-art methods for content extraction. The results can improve content extraction methods for a variety of applications, including search engine optimization and making the web more accessible for the blind or visually impaired.