Towards a Benchmark for Interactive Data Exploration

Existing benchmarks for analytical database systems such as TPC-DS and TPC-H are designed for static reporting scenarios. The main metric of these benchmarks is the performance of running different SQL queries over a predefined database. In this paper, we argue that such benchmarks are not suitable for evaluating modern interactive data exploration (IDE) systems, which allow data scientists of varying skill levels to manipulate, analyze, and explore large data sets, as well as to build models and apply machine learning at interactive speeds. While query performance is still important for data exploration, we believe that a much better metric would reflect the number and complexity of insights users gain in a given amount of time. This paper discusses challenges of creating such a metric and presents ideas towards a new benchmark that simulates typical user behavior and allows IDE systems to be compared in a reproducible way.

[1]  Ben Shneiderman,et al.  Response time and display rate in human performance with computers , 1984, CSUR.

[2]  Daniel A. Keim,et al.  Information Visualization and Visual Data Mining , 2002, IEEE Trans. Vis. Comput. Graph..

[3]  A. ADoefaa,et al.  ? ? ? ? f ? ? ? ? ? , 2003 .

[4]  Catherine Plaisant,et al.  The challenge of information visualization evaluation , 2004, AVI.

[5]  Ben Shneiderman,et al.  Strategies for evaluating information visualization tools: multi-dimensional in-depth long-term case studies , 2006, BELIV '06.

[6]  Chris North,et al.  Toward measuring visualization insight , 2006, IEEE Computer Graphics and Applications.

[7]  Zohra Bellahsene,et al.  XBenchMatch: a Benchmark for XML Schema Matching Tools , 2007, VLDB.

[8]  M. Sheelagh T. Carpendale,et al.  Evaluating Information Visualizations , 2008, Information Visualization.

[9]  Tamara Munzner,et al.  A Nested Model for Visualization Design and Validation , 2009, IEEE Transactions on Visualization and Computer Graphics.

[10]  James H. Johnson,et al.  Just Noticeable Difference , 2010 .

[11]  Patrick Marcel,et al.  A survey of query recommendation techniques for data warehouse exploration , 2011, EDA.

[12]  Dominique Brodbeck,et al.  Research directions in data wrangling: Visualizations and transformations for usable and credible data , 2011, Inf. Vis..

[13]  Stephen J. Wright,et al.  Hogwild: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent , 2011, NIPS.

[14]  Jeffrey Heer,et al.  Wrangler: interactive visual specification of data transformation scripts , 2011, CHI.

[15]  Alfons Kemper,et al.  HyPer: A hybrid OLTP&OLAP main memory database system based on virtual memory snapshots , 2011, 2011 IEEE 27th International Conference on Data Engineering.

[16]  S. Oeltze,et al.  Interactive Visual Analysis of Scientific Data , 2012 .

[17]  B. Shneiderman,et al.  Interactive Dynamics for Visual Analysis , 2012 .

[18]  Tim Kraska,et al.  CrowdER: Crowdsourcing Entity Resolution , 2012, Proc. VLDB Endow..

[19]  Pat Hanrahan.  Analytic database technologies for a new kind of user: the data enthusiast , 2012, SIGMOD Conference.

[20]  Monica M. C. Schraefel,et al.  TouchViz: a case study comparing two interfaces for data analytics on tablets , 2013, CHI.

[21]  Purnamrita Sarkar,et al.  Crowdsourced enumeration queries , 2013, 2013 IEEE 29th International Conference on Data Engineering (ICDE).

[22]  Takeshi Imai,et al.  SeeDB: a simple and morphology-preserving optical clearing agent for neuronal circuit reconstruction , 2013, Nature Neuroscience.

[23]  Tim Kraska,et al.  MLI: An API for Distributed Machine Learning , 2013, 2013 IEEE 13th International Conference on Data Mining.

[24]  Jeffrey Heer,et al.  imMens: Real‐time Visual Querying of Big Data , 2013, Comput. Graph. Forum.

[25]  Tilmann Rabl,et al.  TPC-DI: The First Industry Benchmark for Data Integration , 2014, Proc. VLDB Endow..

[26]  Arnab Nandi,et al.  Distributed and interactive cube exploration , 2014, 2014 IEEE 30th International Conference on Data Engineering.

[27]  Jeffrey Heer,et al.  The Effects of Interactive Latency on Exploratory Visual Analysis , 2014, IEEE Transactions on Visualization and Computer Graphics.

[28]  Arnab Nandi,et al.  Combining User Interaction, Speculative Query Execution and Sampling in the DICE System , 2014, Proc. VLDB Endow..

[29]  Carsten Binnig,et al.  Vizdom: Interactive Analytics through Pen and Touch , 2015, Proc. VLDB Endow..

[30]  Carsten Binnig,et al.  An Architecture for Compiling UDF-centric Workflows , 2015, Proc. VLDB Endow..

[31]  Carsten Binnig,et al.  RODI: A Benchmark for Automatic Mapping Generation in Relational-to-Ontology Data Integration , 2015, ESWC.

[32]  Tim Kraska,et al.  Automating model search for large scale machine learning , 2015, SoCC.

[33]  Ronitt Rubinfeld,et al.  Rapid Sampling for Visualizations with Ordering Guarantees , 2014, Proc. VLDB Endow..

[34]  Tim Kraska,et al.  SampleClean: Fast and Reliable Analytics on Dirty Data , 2015, IEEE Data Eng. Bull..

[35]  Peter Szolovits,et al.  MIMIC-III, a freely accessible critical care database , 2016, Scientific Data.

[36]  Juliana Freire,et al.  Data Polygamy: The Many-Many Relationships among Urban Spatio-Temporal Data Sets , 2016, SIGMOD Conference.

[37]  Carsten Binnig,et al.  VisTrees: fast indexes for interactive data exploration , 2016, HILDA '16.

[38]  David H. Laidlaw,et al.  A Case Study Using Visualization Interaction Logs and Insight Metrics to Understand How Analysts Arrive at Insights , 2016, IEEE Transactions on Visualization and Computer Graphics.

[39]  Carsten Binnig,et al.  The case for interactive data exploration accelerators (IDEAs) , 2016, HILDA '16.

[40]  Tim Kraska,et al.  Toward Sustainable Insights, or Why Polygamy is Bad for You , 2017, CIDR.

[41]  Michael L. Mortensen,et al.  Estimating the Impact of Unknown Unknowns on Aggregate Query Results , 2015, SIGMOD Conference.