Design and implementation of the network video data acquisition system

Big data technology plays a role in promoting the development of Internet video, and data has become an important carrier for users and network video platform to dig the effect of video communication, guide the video production, and master the user's behavior habits. This system establishes the conceptual process from data acquisition, data processing, data visualization. This article makes analysis and research from the following aspects: the URL extraction module, data acquisition module, data processing module, data visualization module and other functions. For the front page, the system adopts HTML+CSS+JavaScript, for data acquisition core module, the system uses DOM to operate the HTML page node to complete data crawling, and uses MVC architecture of CodeIgniter combined with Ajax technology to achieve the separation between the logical layer data acquisition program and application layer access page . For data visualization, Highcharts is used to display the data so that the data has a better interactivity and presentation capabilities.

[1]  Jie Yang,et al.  Mining Chinese social media UGC: a big-data framework for analyzing Douban movie reviews , 2016, Journal of Big Data.

[2]  Zhiqiang Wei,et al.  A Novel Implementation of a Hash Function Based on XML DOM Parser , 2015, 2015 International Conference on Cyber-Enabled Distributed Computing and Knowledge Discovery.

[3]  Tiezheng Nie,et al.  Crawling Result Pages for Data Extraction Based on URL Classification , 2010, 2010 Seventh Web Information Systems and Applications Conference.

[4]  Simon Fong,et al.  A real-time interactive data mining and visualization system using parallel computing , 2015, 2015 Tenth International Conference on Digital Information Management (ICDIM).

[5]  Karuna C. Gull,et al.  Crawling through web to extract the data from Social networking site - Twitter , 2015, 2015 National Conference on Parallel Computing Technologies (PARCOMPTECH).

[6]  Canan Girgin,et al.  Language based web crawling on big data , 2014, 2014 22nd Signal Processing and Communications Applications Conference (SIU).