Twitter Network Mimicking for Data Storage Benchmarking

A significant portion of the textual data available on the Web comes from microblogging services such as Twitter. A considerable body of research has hence investigated methods for processing streams of short texts from such services as well as benchmarking online social networks. However, the costs connected with the acquisition of real microblogs is prohibitive for researchers with limited resources. We address this challenge by proposing TWIG, a benchmark generator for microblogging services similar to Twitter. It is a collection of algorithms to: 1) serialize users and their tweets from Twitter in RDF, 2) analyze such serialized data in RDF to approximate distributions over the underlying social network and 3) mimic a social network by generating synthetic users and tweets based on the approximated distributions. By using TWIG generated data, researchers can carry out preliminary evaluations of social network analysis and NLP approaches at low cost. Experimental and human evaluation results suggest that the synthetic tweets generated by TWIG are hardly distinguishable from human-generated tweets. Moreover, our results also underpin the scalability of our approach. Our Java implementation of TWIG is open-source and can be found at: https://github.com/dice-group/TWIG.