ScatterNet: A Deep Subjective Similarity Model for Visual Analysis of Scatterplots

Similarity measuring methods are widely adopted in a broad range of visualization applications. In this work, we address the challenge of representing human perception in the visual analysis of scatterplots by introducing a novel deep-learning-based approach, ScatterNet, captures perception-driven similarities of such plots. The approach exploits deep neural networks to extract semantic features of scatterplot images for similarity calculation. We create a large labeled dataset consisting of similar and dissimilar images of scatterplots to train the deep neural network. We conduct a set of evaluations including performance experiments and a user study to demonstrate the effectiveness and efficiency of our approach. The evaluations confirm that the learned features capture the human perception of scatterplot similarity effectively. We describe two scenarios to show how ScatterNet can be applied in visual analysis applications.

[1]  Yale Song,et al.  #FluxFlow: Visual Analysis of Anomalous Information Spreading on Social Media , 2014, IEEE Transactions on Visualization and Computer Graphics.

[2]  Jeffrey Heer,et al.  Beyond Weber's Law: A Second Look at Ranking Visualizations of Correlation , 2016, IEEE Transactions on Visualization and Computer Graphics.

[3]  Enrico Bertini,et al.  Quality Metrics in High-Dimensional Data Visualization: An Overview and Systematization , 2011, IEEE Transactions on Visualization and Computer Graphics.

[4]  Aleksandra Mojsilovic,et al.  Capturing image semantics with low-level descriptors , 2001, Proceedings 2001 International Conference on Image Processing (Cat. No.01CH37205).

[5]  Burr Settles,et al.  Active Learning Literature Survey , 2009 .

[6]  Aleksandra Mojsilovic,et al.  ISee: perceptual features for image library navigation , 2002, IS&T/SPIE Electronic Imaging.

[7]  Li Chen,et al.  Cluster-Based Visual Abstraction for Multivariate Scatterplots , 2018, IEEE Transactions on Visualization and Computer Graphics.

[8]  Paul Johns,et al.  Understanding Pen and Touch Interaction for Data Exploration on Interactive Whiteboards , 2012, IEEE Transactions on Visualization and Computer Graphics.

[9]  Michael E. Papka,et al.  Tell me what do you see: Detecting perceptually-separable visual patterns via clustering of image-space features in visualizations , 2015, 2015 IEEE Conference on Visual Analytics Science and Technology (VAST).

[10]  Daniel J. Denis,et al.  The early origins and development of the scatterplot. , 2005, Journal of the history of the behavioral sciences.

[11]  Robert Kosara,et al.  Pargnostics: Screen-Space Metrics for Parallel Coordinates , 2010, IEEE Transactions on Visualization and Computer Graphics.

[12]  Bongshin Lee,et al.  ChartSense: Interactive Data Extraction from Chart Images , 2017, CHI.

[13]  Yoshua Bengio,et al.  How transferable are features in deep neural networks? , 2014, NIPS.

[14]  Lijie Fu,et al.  Implementation of Three-dimensional Scagnostics , 2009 .

[15]  Jean-Daniel Fekete,et al.  Magnostics: Image-Based Search of Interesting Matrix Views for Guided Network Exploration , 2017, IEEE Transactions on Visualization and Computer Graphics.

[16]  Yang Song,et al.  Learning Fine-Grained Image Similarity with Deep Ranking , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[17]  Leland Wilkinson,et al.  ScagExplorer: Exploring Scatterplots by Their Scagnostics , 2014, 2014 IEEE Pacific Visualization Symposium.

[18]  Abhishek Das,et al.  Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization , 2016, 2017 IEEE International Conference on Computer Vision (ICCV).

[19]  Kun Zhou,et al.  Visual Abstraction and Exploration of Multi-class Scatterplots , 2014, IEEE Transactions on Visualization and Computer Graphics.

[20]  Ronald A. Rensink The nature of correlation perception in scatterplots , 2016, Psychonomic bulletin & review.

[21]  Michael S. Bernstein,et al.  Learning Perceptual Kernels for Visualization Design , 2014, IEEE Transactions on Visualization and Computer Graphics.

[22]  Ronald A. Rensink,et al.  The Perception of Correlation in Scatterplots , 2010, Comput. Graph. Forum.

[23]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[24]  Lei Cao,et al.  Online Outlier Exploration Over Large Datasets , 2015, KDD.

[25]  Ching-Yung Lin,et al.  TargetVue: Visual Analysis of Anomalous User Behaviors in Online Communication Systems , 2016, IEEE Transactions on Visualization and Computer Graphics.

[26]  Feng Zhou,et al.  Fine-Grained Categorization and Dataset Bootstrapping Using Deep Metric Learning with Humans in the Loop , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[27]  Ali Farhadi,et al.  FigureSeer: Parsing Result-Figures in Research Papers , 2016, ECCV.

[28]  Steven Franconeri,et al.  Ranking Visualizations of Correlation Using Weber's Law , 2014, IEEE Transactions on Visualization and Computer Graphics.

[29]  Tamara Munzner,et al.  A Taxonomy of Visual Cluster Separation Factors , 2012, Comput. Graph. Forum.

[30]  Geoffrey E. Hinton,et al.  Visualizing Data using t-SNE , 2008 .

[31]  Aleksandra Mojsilovic,et al.  Semantic metric for image library exploration , 2004, IEEE Transactions on Multimedia.

[32]  Daniel F. Keefe,et al.  Visualization-by-Sketching: An Artist's Interface for Creating Multivariate Time-Varying Data Visualizations , 2016, IEEE Transactions on Visualization and Computer Graphics.

[33]  Ben Shneiderman,et al.  A Rank-by-Feature Framework for Interactive Exploration of Multidimensional Data , 2005, Inf. Vis..

[34]  Thomas Schultz,et al.  Open-Box Spectral Clustering: Applications to Medical Image Analysis , 2013, IEEE Transactions on Visualization and Computer Graphics.

[35]  Fatih Korkmaz,et al.  Feedback-driven interactive exploration of large multidimensional data supported by visual classifier , 2014, 2014 IEEE Conference on Visual Analytics Science and Technology (VAST).

[36]  Daniel A. Keim,et al.  Guided Sketching for Visual Search and Exploration in Large Scatter Plot Spaces , 2014, EuroVA@EuroVis.

[37]  Leland Wilkinson,et al.  Transforming Scagnostics to Reveal Hidden Features , 2014, IEEE Transactions on Visualization and Computer Graphics.

[38]  Justin Talbot,et al.  Automatic Selection of Partitioning Variables for Small Multiple Displays , 2016, IEEE Transactions on Visualization and Computer Graphics.

[39]  Marcus A. Magnor,et al.  Selecting Coherent and Relevant Plots in Large Scatterplot Matrices , 2012, Comput. Graph. Forum.

[40]  Jerome H. Friedman,et al.  John W. Tukey's work on interactive graphics , 2002 .

[41]  Meihui Zhang,et al.  Cross-Domain Image Retrieval with Attention Modeling , 2017, ACM Multimedia.

[42]  Marcus A. Magnor,et al.  Perception-based visual quality measures , 2011, 2011 IEEE Conference on Visual Analytics Science and Technology (VAST).

[43]  Daniel A. Keim,et al.  Pixnostics: Towards Measuring the Value of Visualization , 2006, 2006 IEEE Symposium On Visual Analytics Science And Technology.

[44]  Anthony K. H. Tung,et al.  LDSScanner: Exploratory Analysis of Low-Dimensional Structures in High-Dimensional Datasets , 2018, IEEE Transactions on Visualization and Computer Graphics.

[45]  Ross B. Girshick,et al.  Fast R-CNN , 2015, 1504.08083.

[46]  R. Grossman,et al.  Graph-theoretic scagnostics , 2005, IEEE Symposium on Information Visualization, 2005. INFOVIS 2005..

[47]  S. Johansson,et al.  Interactive Dimensionality Reduction Through User-defined Combinations of Quality Metrics , 2009, IEEE Transactions on Visualization and Computer Graphics.

[48]  Daniel A. Keim,et al.  Guiding the Exploration of Scatter Plot Data Using Motif-Based Interest Measures , 2015, 2015 Big Data Visual Analytics (BDVA).

[49]  Anshul Vikram Pandey,et al.  Towards Understanding Human Similarity Perception in the Analysis of Large Sets of Scatter Plots , 2016, CHI.

[50]  Peter Bak,et al.  Visual Analytics for Spatial Clustering: Using a Heuristic Approach for Guided Exploration , 2013, IEEE Transactions on Visualization and Computer Graphics.

[51]  Marcus A. Magnor,et al.  Automated Analytical Methods to Support Visual Exploration of High-Dimensional Data , 2011, IEEE Transactions on Visualization and Computer Graphics.

[52]  James Philbin,et al.  FaceNet: A unified embedding for face recognition and clustering , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[53]  Leland Wilkinson,et al.  TimeSeer: Scagnostics for High-Dimensional Time Series , 2013, IEEE Transactions on Visualization and Computer Graphics.

[54]  David S. Ebert,et al.  DimScanner: A relation-based visual exploration approach towards data dimension inspection , 2016, 2016 IEEE Conference on Visual Analytics Science and Technology (VAST).

[55]  M. Shahriar Hossain,et al.  Scatter/Gather Clustering: Flexibly Incorporating User Feedback to Steer Clustering Results , 2012, IEEE Transactions on Visualization and Computer Graphics.

[56]  John P. Lewis,et al.  Eurographics/ Ieee-vgtc Symposium on Visualization 2009 Selecting Good Views of High-dimensional Data Using Class Consistency , 2022 .

[57]  Jeffrey Heer,et al.  ReVision: automated classification, analysis and redesign of chart images , 2011, UIST.

[58]  Charles A. Bouman,et al.  Perceptual image similarity experiments , 1998, Electronic Imaging.