Insight in Image Collections by Multimedia Pivot Tables

We propose a multimedia analytics solution for getting insight in image collections by extending the powerful method of pivot tables, found in the ubiquitous spreadsheets, to multimedia. Our proposed solution is designed by considering the characteristics of multimedia data as well as insight and provides integral access to visual content through concept detection results, tags, geolocation, and other metadata. We present a set of scenarios of using the pivot tables for a collection of images, tags, and metadata from Flickr. User experiments have been instrumental in realizing the final design presented in this paper. The accompanying video shows the solution in action.

[1]  Daniel A. Keim,et al.  Visual Analytics: Definition, Process, and Challenges , 2008, Information Visualization.

[2]  Laurent Amsaleg,et al.  PhotoCube: effective and efficient multi-dimensional browsing of personal photo collections , 2011, ICMR '11.

[3]  Andreas Paepcke,et al.  PhotoSpread: A Spreadsheet for Managing Photos , 2008, 2008 IEEE 24th International Conference on Data Engineering.

[4]  Marcel Worring,et al.  MediaTable: Interactive Categorization of Multimedia Collections , 2010, IEEE Computer Graphics and Applications.

[5]  D Fisher,et al.  Visualizations everywhere: A Multiplatform Infrastructure for Linked Visualizations , 2010, IEEE Transactions on Visualization and Computer Graphics.

[6]  Colin Ware,et al.  Visual Thinking for Design , 2008 .

[7]  Marcel Worring,et al.  Interactive access to large image collections using similarity-based visualization , 2008, J. Vis. Lang. Comput..

[8]  Jun Ma,et al.  Similarity-based visualization of large image collections , 2015, Inf. Vis..

[9]  E.H. Chi,et al.  Principles for Information Visualization Spreadsheets , 1998, IEEE Computer Graphics and Applications.

[10]  William Ribarsky,et al.  Multimedia Analysis + Visual Analytics = Multimedia Analytics , 2010, IEEE Computer Graphics and Applications.

[11]  Gunther Heidemann,et al.  Inter-active learning of ad-hoc classifiers for video visual analytics , 2012, 2012 IEEE Conference on Visual Analytics Science and Technology (VAST).

[12]  Chris North,et al.  Toward measuring visualization insight , 2006, IEEE Computer Graphics and Applications.

[13]  Marcel Worring,et al.  Multimedia Pivot Tables , 2013 .

[14]  Marcel Worring,et al.  Towards interactive, intelligent, and integrated multimedia analytics , 2014, 2014 IEEE Conference on Visual Analytics Science and Technology (VAST).

[15]  Susan T. Dumais,et al.  PivotPaths: Strolling through Faceted Information Spaces , 2012, IEEE Transactions on Visualization and Computer Graphics.

[16]  Jianping Fan,et al.  Integrating multi-modal content analysis and hyperbolic visualization for large-scale news video retrieval and exploration , 2008, Signal Process. Image Commun..

[17]  Russ Burtner,et al.  Interactive visual comparison of multimedia data through type-specific views , 2013, Electronic Imaging.

[18]  Hanspeter Pfister,et al.  LineUp: Visual Analysis of Multi-Attribute Rankings , 2013, IEEE Transactions on Visualization and Computer Graphics.

[19]  Klaus Schöffmann,et al.  3D Storyboards for Interactive Visual Search , 2012, 2012 IEEE International Conference on Multimedia and Expo.

[20]  Michael G. Christel.  Automated Metadata in Multimedia Information Systems: Creation, Refinement, Use in Surrogates, and Evaluation , 2009, Automated Metadata in Multimedia Information Systems.

[21]  Marcel Worring,et al.  Where Is the User in Multimedia Retrieval? , 2012, IEEE Multim..

[22]  M. Sheelagh T. Carpendale,et al.  A Visual Backchannel for Large-Scale Events , 2010, IEEE Transactions on Visualization and Computer Graphics.

[23]  Frank M. Shipman,et al.  Flexible access to photo libraries via time, place, tags, and visual features , 2010, JCDL '10.

[24]  Jianping Fan,et al.  Semantic Image Browser: Bridging Information Visualization with Automated Intelligent Image Analysis , 2006, 2006 IEEE Symposium On Visual Analytics Science And Technology.

[25]  Pat Hanrahan,et al.  Polaris: A System for Query, Analysis, and Visualization of Multidimensional Relational Databases , 2002, IEEE Trans. Vis. Comput. Graph..

[26]  Luis Gustavo Nonato,et al.  Local Affine Multidimensional Projection , 2011, IEEE Transactions on Visualization and Computer Graphics.

[27]  Hwan-Gue Cho,et al.  PHOTOLAND: a new image layout system using spatio-temporal information in digital photos , 2010, SAC '10.

[28]  Hamid Pirahesh,et al.  Data Cube: A Relational Aggregation Operator Generalizing Group-By, Cross-Tab, and Sub-Totals , 1996, Data Mining and Knowledge Discovery.

[29]  Niklas Elmqvist,et al.  Visualization Mosaics for Multivariate Visual Exploration , 2013, Comput. Graph. Forum.

[30]  Shih-Fu Chang,et al.  Visual islands: intuitive browsing of visual search results , 2008, CIVR '08.

[31]  Michael Alexander,et al.  Pivot Table Data Crunching , 2001 .

[32]  Cees G. M. Snoek,et al.  The MediaMill at TRECVID 2013: : Searching concepts, Objects, Instances and events in video , 2013, TRECVID.