Research on Content Extraction of Rich Text Web Pages

Obtaining effective information from Web pages has become a hot topic in the Internet data processing industry. Based on this, this paper studies the extraction of rich text Web pages. First, this paper designs a method to extract rich text page titles and release times. After a large number of data sets statistics, it is found that almost all the body titles are in the title tag, so a tag-based regular matching algorithm is designed. By writing the code to test the algorithm, the average extraction accuracy is more than 80%. For the extraction of release time, a common regular expression is used to match the full text of a web page and filter according to the position difference of each timestamp in the web page. Set the rule to filter the first occurrence of the timestamp is the release time. Test the algorithm, and the average extraction accuracy of the local data set is over 75%. Secondly, this paper designs a method to extract the text of rich text web pages. On the basis of the training data set, the logical regression model, random forest model and support vector machine model were respectively trained through the cross validation of ten folds. Model fusion was performed on the three models to identify the class tag of the body. Again tested on the test data set, the experimental results show that the average accuracy is 95.6%, the recall rate index was 0.948, 0.923, f1 to further determine the web page text in class labels can be 100% extraction. Finally, this paper evaluates the proposed algorithm and proposes corresponding improvements.