Multimedia Pivot Tables for Multimedia Analytics on Image Collections

We propose a multimedia analytics solution for getting insight into image collections by extending the powerful analytic capabilities of pivot tables, found in the ubiquitous spreadsheets, to multimedia. We formalize the concept of multimedia pivot tables and give design rules and methods for the multimodal summarization, structuring, and browsing of the collection based on these tables, all optimized to support an analyst in getting structural and conclusive insights. Our proposed solution provides truly interactive analytics on the visual content of image collections through concept detection results, as well as tags, geolocation, time, and other metadata. We have performed user experiments with novice users on a dataset from Flickr to improve the initial design and with expert users in marketing and multimedia analysis on two domain-specific datasets collected from Instagram. The results show that analysts are indeed capable of deriving structural and conclusive insights using the proposed multimedia analytics solution. On our website, videos of the system in action are available.

[1]  Jun Ma,et al.  Similarity-based visualization of large image collections , 2015, Inf. Vis..

[2]  E.H. Chi,et al.  Principles for Information Visualization Spreadsheets , 1998, IEEE Computer Graphics and Applications.

[3]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[4]  Geoffrey E. Hinton,et al.  Deep Learning , 2015, Nature.

[5]  Gunther Heidemann,et al.  Inter-active learning of ad-hoc classifiers for video visual analytics , 2012, 2012 IEEE Conference on Visual Analytics Science and Technology (VAST).

[6]  Jean-Daniel Fekete,et al.  Hierarchical Aggregation for Information Visualization: Overview, Techniques, and Design Guidelines , 2010, IEEE Transactions on Visualization and Computer Graphics.

[7]  Chris North,et al.  Toward measuring visualization insight , 2006, IEEE Computer Graphics and Applications.

[8]  Andreas Paepcke,et al.  PhotoSpread: A Spreadsheet for Managing Photos , 2008, 2008 IEEE 24th International Conference on Data Engineering.

[9]  Klaus Schöffmann,et al.  Video Interaction Tools , 2015, ACM Comput. Surv..

[10]  Chih-Jen Lin,et al.  Large-Scale Video Summarization Using Web-Image Priors , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[11]  Fei-Fei Li,et al.  What Does Classifying More Than 10, 000 Image Categories Tell Us? , 2010, ECCV.

[12]  Rongrong Ji,et al.  Large-scale visual sentiment ontology and detectors using adjective noun pairs , 2013, ACM Multimedia.

[13]  D Fisher,et al.  Visualizations everywhere: A Multiplatform Infrastructure for Linked Visualizations , 2010, IEEE Transactions on Visualization and Computer Graphics.

[14]  Daniel A. Keim,et al.  Visual Analytics: Definition, Process, and Challenges , 2008, Information Visualization.

[15]  Marcel Worring,et al.  Interactive access to large image collections using similarity-based visualization , 2008, J. Vis. Lang. Comput..

[16]  Shih-Fu Chang,et al.  Visual islands: intuitive browsing of visual search results , 2008, CIVR '08.

[17]  Hwan-Gue Cho,et al.  PHOTOLAND: a new image layout system using spatio-temporal information in digital photos , 2010, SAC '10.

[18]  Hanspeter Pfister,et al.  LineUp: Visual Analysis of Multi-Attribute Rankings , 2013, IEEE Transactions on Visualization and Computer Graphics.

[19]  Klaus Schöffmann,et al.  3D Storyboards for Interactive Visual Search , 2012, 2012 IEEE International Conference on Multimedia and Expo.

[20]  Qing Chen,et al.  PeakVizor: Visual Analytics of Peaks in Video Clickstreams from Massive Open Online Courses , 2016, IEEE Transactions on Visualization and Computer Graphics.

[21]  Jarke J. van Wijk,et al.  ICLIC: Interactive categorization of large image collections , 2016, 2016 IEEE Pacific Visualization Symposium (PacificVis).

[22]  Marcel Worring,et al.  Active Bucket Categorization for High Recall Video Retrieval , 2013, IEEE Transactions on Multimedia.

[23]  Qi Tian,et al.  Multimedia search reranking: A literature survey , 2014, CSUR.

[24]  Marcel Worring,et al.  MediaTable: Interactive Categorization of Multimedia Collections , 2010, IEEE Computer Graphics and Applications.

[25]  Daniel A. Keim,et al.  Mastering the Information Age - Solving Problems with Visual Analytics , 2010 .

[26]  Hamid Pirahesh,et al.  Data Cube: A Relational Aggregation Operator Generalizing Group-By, Cross-Tab, and Sub-Totals , 1996, Data Mining and Knowledge Discovery.

[27]  Cees G. M. Snoek,et al.  The MediaMill at TRECVID 2013: : Searching concepts, Objects, Instances and events in video , 2013, TRECVID.

[28]  Pat Hanrahan,et al.  Polaris: a system for query, analysis and visualization of multi-dimensional relational databases , 2000, IEEE Symposium on Information Visualization 2000. INFOVIS 2000. Proceedings.

[29]  Frank M. Shipman,et al.  Flexible access to photo libraries via time, place, tags, and visual features , 2010, JCDL '10.

[30]  Marcel Worring,et al.  Insight in Image Collections by Multimedia Pivot Tables , 2015, ICMR.

[31]  Colin Ware,et al.  Visual Thinking for Design , 2008 .

[32]  Andreas Kerren,et al.  Text visualization techniques: Taxonomy, visual survey, and community insights , 2015, 2015 IEEE Pacific Visualization Symposium (PacificVis).

[33]  William Ribarsky,et al.  Multimedia Analysis + Visual Analytics = Multimedia Analytics , 2010, IEEE Computer Graphics and Applications.

[34]  Eric P. Xing,et al.  Joint Summarization of Large-Scale Collections of Web Images and Videos for Storyline Reconstruction , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[35]  Ba Tu Truong,et al.  Video abstraction: A systematic review and classification , 2007, TOMCCAP.

[36]  Laurent Amsaleg,et al.  PhotoCube: effective and efficient multi-dimensional browsing of personal photo collections , 2011, ICMR '11.

[37]  Niklas Elmqvist,et al.  Visualization Mosaics for Multivariate Visual Exploration , 2013, Comput. Graph. Forum.

[38]  Marcel Worring,et al.  Towards interactive, intelligent, and integrated multimedia analytics , 2014, 2014 IEEE Conference on Visual Analytics Science and Technology (VAST).

[39]  Jianping Fan,et al.  Integrating multi-modal content analysis and hyperbolic visualization for large-scale news video retrieval and exploration , 2008, Signal Process. Image Commun..

[40]  Russ Burtner,et al.  Interactive visual comparison of multimedia data through type-specific views , 2013, Electronic Imaging.

[41]  Andreas Kerren,et al.  Text Visualization Browser : A Visual Survey of Text Visualization Techniques , 2014 .

[42]  Jianping Fan,et al.  Semantic Image Browser: Bridging Information Visualization with Automated Intelligent Image Analysis , 2006, 2006 IEEE Symposium On Visual Analytics Science And Technology.

[43]  Marcel Worring,et al.  Ten Research Questions for Scalable Multimedia Analytics , 2016, MMM.

[44]  Heidrun Schumann,et al.  Visualization of Time-Oriented Data , 2011, Human-Computer Interaction Series.

[45]  Luis Gustavo Nonato,et al.  Local Affine Multidimensional Projection , 2011, IEEE Transactions on Visualization and Computer Graphics.

[46]  Chris North,et al.  A comparison of benchmark task and insight evaluation methods for information visualization , 2011, Inf. Vis..

[47]  Michael Alexander,et al.  Pivot Table Data Crunching , 2001 .

[48]  Marcel Worring,et al.  Content-Based Image Retrieval at the End of the Early Years , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[49]  Michael G. Christel.  Automated Metadata in Multimedia Information Systems: Creation, Refinement, Use in Surrogates, and Evaluation , 2009, Automated Metadata in Multimedia Information Systems.

[50]  Kristin A. Cook,et al.  Illuminating the Path: The Research and Development Agenda for Visual Analytics , 2005 .