RevDet: Robust and Memory Efficient Event Detection and Tracking in Large News Feeds

With the ever-growing volume of online news feeds, event-based organization of news articles has many practical applications including better information navigation and the ability to view and analyze events as they develop. Automatically tracking the evolution of events in large news corpora still remains a challenging task, and the existing techniques for Event Detection and Tracking do not place a particular focus on tracking events in very large and constantly updating news feeds. Here, we propose a new method for robust and efficient event detection and tracking, which we call RevDet algorithm. RevDet adopts an iterative clustering approach for tracking events. Even though many events continue to develop for many days or even months, RevDet is able to detect and track those events while utilizing only a constant amount of space on main memory.We also devise a redundancy removal strategy which effectively eliminates duplicate news articles and substantially reduces the size of data. We construct a large, comprehensive new ground truth dataset specifically for event detection and tracking approaches by augmenting two existing datasets: w2e and GDELT. We implement RevDet algorithm and evaluate its performance on the ground truth event chains. We discover that our algorithm is able to accurately recover event chains in the ground-truth dataset. We also compare the memory efficiency of our algorithm with the standard single pass clustering approach, and demonstrate the appropriateness of our algorithm for event detection and tracking task in large news feeds.

[1]  Hila Becker,et al.  Beyond Trending Topics: Real-World Event Identification on Twitter , 2011, ICWSM.

[2]  Zahir Tari,et al.  A Survey of Clustering Algorithms for Big Data: Taxonomy and Empirical Analysis , 2014, IEEE Transactions on Emerging Topics in Computing.

[3]  С.О. Грищенко.  2014 , 2019, The Winning Cars of the Indianapolis 500.

[4]  Quoc V. Le,et al.  Distributed Representations of Sentences and Documents , 2014, ICML.

[5]  Tuan-Anh Hoang,et al.  W2E , 2018, Proceedings of the 27th ACM International Conference on Information and Knowledge Management.

[6]  Iadh Ounis,et al.  Real-Time Detection, Tracking, and Monitoring of Automatically Discovered Events in Social Media , 2014, ACL.

[7]  Yukio Ohsawa,et al.  KeyGraph: automatic indexing by co-occurrence graph based on building construction metaphor , 1998, Proceedings IEEE International Forum on Research and Technology Advances in Digital Libraries -ADL'98-.

[8]  مسعود رسول آبادی,et al.  2011 , 2012, The Winning Cars of the Indianapolis 500.

[9]  Yiming Yang,et al.  Topic Detection and Tracking Pilot Study Final Report , 1998 .

[10]  R. Papka,et al.  On-line new event detection and tracking , 1998, SIGIR '98.

[11]  Xun Wang,et al.  Real Time Event Detection in Twitter , 2013, WAIM.

[12]  Duc-Thuan Vo,et al.  Exploiting Language Models to Classify Events from Twitter , 2015, Comput. Intell. Neurosci..

[13]  Kalina Bontcheva,et al.  Sub-story detection in Twitter with hierarchical Dirichlet processes , 2016, Inf. Process. Manag..

[14]  Mehmet A. Orgun,et al.  Real-time event detection from the Twitter data stream using the TwitterNews+ Framework , 2019, Inf. Process. Manag..

[15]  Bo Huang,et al.  Microblog Topic Detection Based on LDA Model and Single-Pass Clustering , 2012, RSCTC.

[16]  Gaël Varoquaux,et al.  Scikit-learn: Machine Learning in Python , 2011, J. Mach. Learn. Res..

[17]  Tian Zhang,et al.  BIRCH: an efficient data clustering method for very large databases , 1996, SIGMOD '96.

[18]  Matthew Hurst,et al.  Event Detection and Tracking in Social Streams , 2009, ICWSM.

[19]  Wolfgang Nejdl,et al.  W2E: A Worldwide-Event Benchmark Dataset for Topic Detection and Tracking , 2018, CIKM.

[20]  Ee-Peng Lim,et al.  Finding Bursty Topics from Microblogs , 2012, ACL.

[21]  Birch , 2020, The Long, Long Life of Trees.

[22]  Eric Horvitz,et al.  Mining the web to predict future events , 2013, WSDM.

[23]  Jure Leskovec,et al.  Meme-tracking and the dynamics of the news cycle , 2009, KDD.