Social multimedia: highlighting opportunities for search and mining of multimedia data in social media applications

In recent years, various Web-based sharing and community services such as Flickr and YouTube have made a vast and rapidly growing amount of multimedia content available online. Uploaded by individual participants, content in these immense pools of content is accompanied by varied types of metadata, such as social network data or descriptive textual information. These collections present, at once, new challenges and exciting opportunities for multimedia research. This article presents an approach for “social multimedia” applications. The approach is based on the experience of building a number of successful applications that are based on mining multimedia content analysis in social multimedia context.

[1]  John L. Arnott,et al.  Interface metaphor design and instant messaging for older adults , 2008, CHI Extended Abstracts.

[2]  Jonathan Goldstein,et al.  When Is ''Nearest Neighbor'' Meaningful? , 1999, ICDT.

[3]  Oded Nov,et al.  What drives content tagging: the case of photos on Flickr , 2008, CHI.

[4]  A. Smeaton,et al.  Combination of content analysis and context features for digital photograph retrieval. , 2005 .

[5]  Wei-Ying Ma,et al.  Multimedia information retrieval: what is it, and why isn't anyone using it? , 2005, MIR '05.

[6]  Yang Song,et al.  Tour the world: Building a web-scale landmark recognition engine , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[7]  Avery Wang,et al.  An Industrial Strength Audio Search Algorithm , 2003, ISMIR.

[8]  K. Perreault,et al.  Research Design: Qualitative, Quantitative, and Mixed Methods Approaches , 2011 .

[9]  Alan Hanjalic,et al.  The Multimedian Concert-Video Browser , 2005, 2005 IEEE International Conference on Multimedia and Expo.

[10]  Thomas Sporer,et al.  PEAQ - The ITU Standard for Objective Measurement of Perceived Audio Quality , 2000 .

[11]  Shih-Fu Chang,et al.  Image Retrieval: Current Techniques, Promising Directions, and Open Issues , 1999, J. Vis. Commun. Image Represent..

[12]  Mor Naaman,et al.  Why we tag: motivations for annotation in mobile and online media , 2007, CHI.

[13]  Susanne Boll MultiTube--Where Web 2.0 and Multimedia Could Meet , 2007, IEEE MultiMedia.

[14]  Jiebo Luo,et al.  Inferring generic activities and events from image content and bags of geo-tags , 2008, CIVR '08.

[15]  Xian-Sheng Hua,et al.  Learning semantic distance from community-tagged media collection , 2009, MM '09.

[16]  Dan R. Olsen,et al.  Evaluating user interface systems research , 2007, UIST.

[17]  David G. Stork,et al.  Pattern Classification (2nd ed.) , 1999 .

[18]  Nenghai Yu,et al.  Distance metric learning from uncertain side information with application to automated photo tagging , 2009, ACM Multimedia.

[19]  Wei-Ying Ma,et al.  VirtualTour: an online travel assistant based on high quality images , 2006, MM '06.

[20]  Ramesh C. Jain,et al.  Classification and annotation of digital photos using optical context data , 2008, CIVR '08.

[21]  Meng Wang,et al.  Visual tag dictionary: interpreting tags with visual words , 2009, WSMC '09.

[22]  Andreas Paepcke,et al.  Time as essence for photo browsing through personal digital libraries , 2002, JCDL '02.

[23]  Roelof van Zwol,et al.  Flickr tag recommendation based on collective knowledge , 2008, WWW.

[24]  Mor Naaman,et al.  Photos on the go: a mobile application case study , 2008, CHI.

[25]  B. Paillard,et al.  PERCEVAL: Perceptual Evaluation of the Quality of Audio Signals , 1992 .

[26]  Shingo Uchihashi,et al.  Video Manga: generating semantically meaningful video summaries , 1999, MULTIMEDIA '99.

[27]  Munmun De Choudhury,et al.  What makes conversations interesting?: themes, participants and consequences of conversations in online social media , 2009, WWW '09.

[28]  Marcel Worring,et al.  Content-Based Image Retrieval at the End of the Early Years , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[29]  Natasha Gelfand,et al.  Visual summaries of popular landmarks from community photo collections , 2009 .

[30]  Mor Naaman,et al.  Over-exposed?: privacy patterns and considerations in online and mobile photo sharing , 2007, CHI.

[31]  Edward Y. Chang,et al.  Multimodal metadata fusion using causal strength , 2005, ACM Multimedia.

[32]  Yiming Liu,et al.  Enhancing online personal connections through the synchronized sharing of online video , 2008, CHI Extended Abstracts.

[33]  Jiebo Luo,et al.  Pictures are not taken in a vacuum - an overview of exploiting context for semantic scene content understanding , 2006, IEEE Signal Processing Magazine.

[34]  Hila Becker,et al.  Learning similarity metrics for event identification in social media , 2010, WSDM '10.

[35]  Ramesh C. Jain,et al.  Between context-aware media capture and multimedia content analysis: where do we find the promised land? , 2004, MULTIMEDIA '04.

[36]  David A. Shamma,et al.  Tweet the debates: understanding community annotation of uncollected sources , 2009, WSM@MM.

[37]  Fred Stentiford,et al.  Using context and similarity for face and location identification , 2006, Electronic Imaging.

[38]  Alexander C. Loui,et al.  Event-based location matching for consumer image collections , 2008, CIVR '08.

[39]  Mor Naaman,et al.  ZoneTag's Collaborative Tag Suggestions: What is This Person Doing in My Phone? , 2008, IEEE MultiMedia.

[40]  Cees Snoek,et al.  Can social tagged images aid concept-based video search? , 2009, 2009 IEEE International Conference on Multimedia and Expo.

[41]  Djemel Ziou,et al.  A Graphical Model for Context-Aware Visual Content Recommendation , 2008, IEEE Transactions on Multimedia.

[42]  Simon King,et al.  From context to content: leveraging context to infer media metadata , 2004, MULTIMEDIA '04.

[43]  Tobun Dorbin Ng,et al.  Collages as dynamic summaries for news video , 2002, MULTIMEDIA '02.

[44]  John W. Creswell,et al.  Research Design: Qualitative, Quantitative, and Mixed Methods Approaches , 2010 .

[45]  B. Bergum,et al.  Attention and performance IX , 1982 .

[46]  Ton Kalker,et al.  A Highly Robust Audio Fingerprinting System With an Efficient Search Strategy , 2003 .

[47]  Hans Weda,et al.  Synchronization of multiple video recordings based on still camera flashes , 2006, MM '06.

[48]  Claudio Gutierrez,et al.  Survey of graph database models , 2008, CSUR.

[49]  Z. Meral Özsoyoglu,et al.  Annotation suggestion and search for personal multimedia objects on the web , 2008, CIVR '08.

[50]  Patrick Schmitz,et al.  Community annotation and remix: a research platform and pilot deployment , 2006, HCM '06.

[51]  Mor Naaman,et al.  World explorer: visualizing aggregate data from unstructured text in geo-referenced collections , 2007, JCDL '07.

[52]  Jiebo Luo,et al.  Annotating photo collections by label propagation according to multiple similarity cues , 2008, ACM Multimedia.

[53]  Alan F. Smeaton,et al.  Context-Aware Person Identification in Personal Photo Collections , 2009, IEEE Transactions on Multimedia.

[54]  Mor Naaman,et al.  Summarization of online image collections via implicit feedback , 2007, WWW '07.

[55]  Tanveer F. Syeda-Mahmood,et al.  Learning video browsing behavior and its application in the generation of video previews , 2001, MULTIMEDIA '01.

[56]  B. Shneiderman Science 2.0 , 2008, Science.

[57]  Shuicheng Yan,et al.  Inferring semantic concepts from community-contributed images and noisy tags , 2009, ACM Multimedia.

[58]  James Ze Wang,et al.  Image retrieval: Ideas, influences, and trends of the new age , 2008, CSUR.

[59]  Martin Halvey,et al.  WWW '07: Proceedings of the 16th international conference on World Wide Web , 2007, WWW 2007.

[60]  Mor Naaman,et al.  How flickr helps us make sense of the world: context and content in community-contributed media collections , 2007, ACM Multimedia.

[61]  Mohand Boughanem,et al.  Proceedings of the 31th European Conference on IR Research on Advances in Information Retrieval , 2009 .

[62]  Mor Naaman,et al.  Towards automatic extraction of event and place semantics from flickr tags , 2007, SIGIR.

[63]  Kentaro Toyama,et al.  Geographic location tags on digital images , 2003, ACM Multimedia.

[64]  Marcel Worring,et al.  The Role of Visual Content and Style for Concert Video Indexing , 2007, 2007 IEEE International Conference on Multimedia and Expo.

[65]  Mohan S. Kankanhalli,et al.  Proceedings of the 2008 international conference on Content-based image and video retrieval , 2008 .

[66]  John G. Beerends,et al.  A Perceptual Audio Quality Measure Based on a Psychoacoustic Sound Representation , 1992 .

[67]  Benoit Huet,et al.  Proceedings of the 1st workshop on Web-scale multimedia corpus , 2009, MM 2009.

[68]  Dinh Q. Phung,et al.  Flickr hypergroups , 2009, ACM Multimedia.

[69]  Henning Schulzrinne,et al.  Proceedings of the 12th annual ACM international conference on Multimedia , 2004, MM 2004.

[70]  Nenghai Yu,et al.  Flickr distance , 2008, ACM Multimedia.

[71]  Steven M. Seitz,et al.  Finding paths through the world's photos , 2008, SIGGRAPH 2008.

[72]  Mor Naaman,et al.  From Where to What: Metadata Sharing for Digital Photographs with Geographic Coordinates , 2003, OTM.

[73]  Shigeo Abe DrEng Pattern Classification , 2001, Springer London.

[74]  Nick Reid,et al.  Photo LOI: browsing multi-user photo collections , 2005, MULTIMEDIA '05.

[75]  Alan F. Smeaton,et al.  My digital photos: where and when? , 2005, MULTIMEDIA '05.

[76]  Xing Xie,et al.  Mining city landmarks from blogs by graph modeling , 2009, ACM Multimedia.

[77]  Edward Y. Chang,et al.  Extent: Inferring Image Metadata from Context and Content , 2005, 2005 IEEE International Conference on Multimedia and Expo.

[78]  Tamara L. Berg,et al.  Automatic Ranking of Iconic Images , 2007 .

[79]  David M. Nichols,et al.  How people find videos , 2008, JCDL '08.

[80]  Ramesh Jain,et al.  Toward a Common Event Model for Multimedia Applications , 2007, IEEE MultiMedia.

[81]  Nicu Sebe,et al.  Content-based multimedia information retrieval: State of the art and challenges , 2006, TOMCCAP.

[82]  John P. Oakley,et al.  Storage and Retrieval for Image and Video Databases , 1993 .

[83]  Mor Naaman,et al.  Generating diverse and representative image search results for landmarks , 2008, WWW.

[84]  Alan F. Smeaton,et al.  MediAssist: Using Content-Based Analysis and Context to Manage Personal Photo Collections , 2006, CIVR.

[85]  Mor Naaman,et al.  Generating summaries and visualization for large collections of geo-referenced photographs , 2006, MIR '06.

[86]  Ying Liu,et al.  A survey of content-based image retrieval with high-level semantics , 2007, Pattern Recognit..

[87]  Susanne Boll,et al.  MetaXa - Context- and Content-Driven Metadata Enhancement for Personal Photo Books , 2007, MMM.

[88]  Hila Becker,et al.  Event Identification in Social Media , 2009, WebDB.

[89]  Antti Oulasvirta,et al.  Collective creation and sense-making of mobile media , 2006, CHI.

[90]  Dick C. A. Bulterman,et al.  Is It Time for a Moratorium on Metadata? , 2004, IEEE Multim..

[91]  Alan Hanjalic,et al.  Intelligent browsing of concert videos , 2007, ACM Multimedia.

[92]  Matthijs C. Dorst Distinctive Image Features from Scale-Invariant Keypoints , 2011 .

[93]  H. Garcia-Molina,et al.  Automatic organization for digital photographs with geographic coordinates , 2004, Proceedings of the 2004 Joint ACM/IEEE Conference on Digital Libraries, 2004..

[94]  Daniel Gatica-Perez,et al.  Analyzing Flickr groups , 2008, CIVR '08.

[95]  P. Schmitz,et al.  Inducing Ontology from Flickr Tags , 2006 .

[96]  Peter Brusilovsky,et al.  Social navigation in web lectures , 2006, HYPERTEXT '06.

[97]  Edward Y. Chang Organizing multimedia data socially , 2008, CIVR '08.

[98]  Susanne Boll,et al.  Semantics, content, and structure of many for the creation of personal photo albums , 2007, ACM Multimedia.

[99]  Shih-Fu Chang,et al.  To search or to label?: predicting the performance of search-based automatic image classifiers , 2006, MIR '06.

[100]  B. S. Manjunath,et al.  Texture Features for Browsing and Retrieval of Image Data , 1996, IEEE Trans. Pattern Anal. Mach. Intell..

[101]  Mor Naaman,et al.  Leveraging context to resolve identity in photo albums , 2005, Proceedings of the 5th ACM/IEEE-CS Joint Conference on Digital Libraries (JCDL '05).

[102]  Steffen Staab,et al.  Exploiting Flickr Tags and Groups for Finding Landmark Photos , 2009, ECIR.

[103]  Jiang-Ming Yang,et al.  Generating location overviews with images and tags by mining user-generated travelogues , 2009, ACM Multimedia.

[104]  Mor Naaman,et al.  Automatically generating metadata for digital photographs with geographic coordinates , 2004, WWW Alt. '04.

[105]  Markus A. Stricker,et al.  Similarity of color images , 1995, Electronic Imaging.

[106]  Darren Gergle,et al.  Emotion rating from short blog texts , 2008, CHI.

[107]  Edward Y. Chang,et al.  EXTENT: fusing context, content, and semantic ontology for photo annotation , 2005, CVDB '05.

[108]  Scott P. Robertson,et al.  Proceedings of the SIGCHI Conference on Human Factors in Computing Systems , 1991 .

[109]  Mor Naaman,et al.  Context data in geo-referenced digital photo collections , 2004, MULTIMEDIA '04.

[110]  Ravi Kumar,et al.  Visualizing tags over time , 2006, WWW '06.

[111]  Lexing Xie,et al.  Contextual wisdom: social relations and correlations for multimedia event annotation , 2007, ACM Multimedia.

[112]  Shih-Fu Chang,et al.  A reranking approach for context-based concept fusion in video indexing and retrieval , 2007, CIVR '07.

[113]  Mor Naaman,et al.  Less talk, more rock: automated organization of community-contributed collections of concert videos , 2009, WWW '09.

[114]  Mauro Barbieri,et al.  Synchronization of multi-camera video recordings based on audio , 2007, ACM Multimedia.

[115]  Jon M. Kleinberg,et al.  Mapping the world's photos , 2009, WWW '09.

[116]  Fan Zi-zhu A Survey of Content-based Image Retrieval , 2005 .

[117]  David A. Shamma,et al.  Watch what I watch: using community activity to understand content , 2007, MIR '07.

[118]  Peter J. Nürnberg,et al.  Proceedings of the seventeenth conference on Hypertext and hypermedia , 2006 .

[119]  Steven M. Seitz,et al.  Scene Summarization for Online Image Collections , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[120]  Marc Gelgon,et al.  Organizing a personal image collection with statistical model-based ICL clustering on spatio-temporal camera phone meta-data , 2004, Journal of Visual Communication and Image Representation.

[121]  Svetha Venkatesh,et al.  Extraction of social context and application to personal multimedia exploration , 2006, MM '06.