Event-based cross media question answering

User generated content, available in massive amounts on the Internet, is receiving increased attention due to its many potential applications. One of such applications is the representation of events using multimedia data. In this paper, an event-based cross media question answering system, which retrieves and summarizes events on a given topic is proposed. In other words, we present a framework for leveraging social media data to extract and illustrate social events automatically on any given query. The system is built in three steps. First, the input query is parsed semantically to identify the topic, location, and time information related to the News of interest. Then, we use the parsed information to mine the latest and hottest related News from social news web services. Third, to identify a unique event, we model the News content by latent Dirichlet Allocation and cluster the News using the DBSCAN algorithm. In the end, for each event, we retrieve both textual and visual content of News that refer the same event. The resulting documents are shown within a vivid interface featuring both event description, tag cloud and photo collage.

[1]  Meng Wang,et al.  Detecting Group Activities With Multi-Camera Context , 2013, IEEE Transactions on Circuits and Systems for Video Technology.

[2]  Yi Wang,et al.  Looking into the world on Google Maps with view direction estimated photos , 2012, Neurocomputing.

[3]  Jitendra Malik,et al.  Normalized cuts and image segmentation , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[4]  Hila Becker,et al.  Identifying content for planned events across social media sites , 2012, WSDM '12.

[5]  Guan Yi,et al.  A Statistical Approach for Content Extraction from Web Page , 2004 .

[6]  Yutaka Matsuo,et al.  Earthquake shakes Twitter users: real-time event detection by social sensors , 2010, WWW '10.

[7]  Sergei Vassilvitskii,et al.  k-means++: the advantages of careful seeding , 2007, SODA '07.

[8]  Tao Mei,et al.  Video collage: presenting a video sequence using a single image , 2008, The Visual Computer.

[9]  Bernard Mérialdo,et al.  Comparison of multi-episode video summarisation algorithms , 2001, 2001 IEEE Fourth Workshop on Multimedia Signal Processing (Cat. No.01TH8564).

[10]  Bu-Sung Lee,et al.  Event Detection in Twitter , 2011, ICWSM.

[11]  Raphaël Troncy,et al.  EURECOM @ MediaEval 2011 Social Event Detection Task , 2011, MediaEval.

[12]  Yue Gao,et al.  Beyond Text QA: Multimedia Answer Generation by Harvesting Web Information , 2013, IEEE Transactions on Multimedia.

[13]  Luc Van Gool,et al.  World-scale mining of objects and events from community photo collections , 2008, CIVR '08.

[14]  Meng Wang,et al.  Event Driven Web Video Summarization by Tag Localization and Key-Shot Identification , 2012, IEEE Transactions on Multimedia.

[15]  Meng Wang,et al.  Multimedia Question Answering , 2010, IEEE MultiMedia.

[16]  Yiming Yang,et al.  Topic Detection and Tracking Pilot Study Final Report , 1998 .

[17]  Ling Chen,et al.  Event detection from flickr data through wavelet-based spatial analysis , 2009, CIKM.

[18]  Xiaojin Zhu,et al.  A Text-to-Picture Synthesis System for Augmenting Communication , 2007, AAAI.

[19]  Raphaël Troncy,et al.  Finding media illustrating events , 2011, ICMR '11.

[20]  Changsheng Xu,et al.  Multimedia news digger on emerging topics from social streams , 2012, ACM Multimedia.

[21]  Tat-Seng Chua,et al.  Video reference: question answering on YouTube , 2009, MM '09.

[22]  Xuelong Li,et al.  Visual-Textual Joint Relevance Learning for Tag-Based Social Image Search , 2013, IEEE Transactions on Image Processing.

[23]  Hinrich Schütze,et al.  Introduction to information retrieval , 2008 .

[24]  Prasenjit Mitra,et al.  Event detection with spatial latent Dirichlet allocation , 2011, JCDL '11.

[25]  João Magalhães,et al.  Assisted news reading with automated illustration , 2010, ACM Multimedia.

[26]  Bernard Mérialdo,et al.  Comparison of Multiepisode Video Summarization Algorithms , 2003, EURASIP J. Adv. Signal Process..

[27]  Meng Wang,et al.  Visual query suggestion , 2010, ACM Trans. Multim. Comput. Commun. Appl..

[28]  Yue Gao,et al.  When Amazon Meets Google: Product Visualization by Exploring Multiple Web Sources , 2013, TOIT.

[29]  Wolfgang Nejdl,et al.  Bringing order to your photos: event-driven classification of flickr images based on social knowledge , 2010, CIKM.

[30]  James Ze Wang,et al.  The Story Picturing Engine---a system for automatic text illustration , 2006, TOMCCAP.