Controlling False Discoveries During Interactive Data Exploration

Recent tools for interactive data exploration significantly increase the chance that users make false discoveries. They allow users to (visually) examine many hypotheses and make inference with simple interactions, and thus incur the issue commonly known in statistics as the "multiple hypothesis testing error." In this work, we propose a solution to integrate the control of multiple hypothesis testing into interactive data exploration systems. A key insight is that existing methods for controlling the false discovery rate (such as FDR) are not directly applicable to interactive data exploration. We therefore discuss a set of new control procedures that are better suited for this task and integrate them in our system, QUDE. Via extensive experiments on both real-world and synthetic data sets we demonstrate how QUDE can help experts and novice users alike to efficiently control false discoveries.

[1]  L. M. M.-T..  Theory of Probability , 1929, Nature.

[2]  J. I.  The Design of Experiments , 1936, Nature.

[3]  J. Neyman,et al.  Consistent Estimates Based on Partially Consistent Observations , 1948 .

[4]  Z. Šidák.  Rectangular Confidence Regions for the Means of Multivariate Normal Distributions , 1967 .

[5]  S. Holm.  A Simple Sequentially Rejective Multiple Test Procedure , 1979 .

[6]  R. F. Wagner,et al.  Efficiency of human visual signal discrimination. , 1981, Science.

[7]  M. Schemper.  A survey of permutation tests for censored survival data , 1984 .

[8]  R. Simes,et al.  An improved Bonferroni procedure for multiple tests of significance , 1986 .

[9]  Y. Hochberg.  A sharper Bonferroni procedure for multiple tests of significance , 1988 .

[10]  Ron Kohavi,et al.  A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection , 1995, IJCAI.

[11]  J. Shaffer.  Multiple Hypothesis Testing , 1995 .

[12]  Y. Benjamini,et al.  Controlling the false discovery rate: a practical and powerful approach to multiple testing , 1995 .

[13]  D. Berry,et al.  Bayesian perspectives on multiple comparisons , 1999 .

[14]  Y. Benjamini,et al.  THE CONTROL OF THE FALSE DISCOVERY RATE IN MULTIPLE TESTING UNDER DEPENDENCY , 2001 .

[15]  J. Ioannidis.  Why Most Published Research Findings Are False , 2005, PLoS medicine.

[16]  Janez Demsar,et al.  Statistical Comparisons of Classifiers over Multiple Data Sets , 2006, J. Mach. Learn. Res..

[17]  Jarke J. van Wijk,et al.  Supporting the analytical reasoning process in information visualization , 2008, CHI.

[18]  Dean P. Foster,et al.  α‐investing: a procedure for sequential control of expected false discoveries , 2008 .

[19]  Alain F. Zuur,et al.  A protocol for data exploration to avoid common statistical problems , 2010 .

[20]  Daniel Perry,et al.  VizDeck: self-organizing dashboards for visual analytics , 2012, SIGMOD Conference.

[21]  Pat Hanrahan.  Analytic database technologies for a new kind of user: the data enthusiast , 2012, SIGMOD Conference.

[22]  R. Tibshirani,et al.  Sequential selection procedures and false discovery rate control , 2013, 1309.5352.

[23]  Jeffrey Heer,et al.  imMens: Real‐time Visual Querying of Big Data , 2013, Comput. Graph. Forum.

[24]  S. Rosset,et al.  Generalized α‐investing: definitions, optimality results and application to public databases , 2014 .

[25]  Arnab Nandi,et al.  Distributed and interactive cube exploration , 2014, 2014 IEEE 30th International Conference on Data Engineering.

[26]  Tyler Cymet,et al.  The era of big data. , 2014, Maryland medicine : MM : a publication of MEDCHI, the Maryland State Medical Society.

[27]  Aditya G. Parameswaran,et al.  SeeDB: Efficient Data-Driven Visualization Recommendations to Support Visual Analytics , 2015, Proc. VLDB Endow..

[28]  Carsten Binnig,et al.  Vizdom: Interactive Analytics through Pen and Touch , 2015, Proc. VLDB Endow..

[29]  Avrim Blum,et al.  The Ladder: A Reliable Leaderboard for Machine Learning Competitions , 2015, ICML.

[30]  Toniann Pitassi,et al.  Preserving Statistical Validity in Adaptive Data Analysis , 2014, STOC.

[31]  Kanit Wongsuphasawat,et al.  Voyager: Exploratory Analysis via Faceted Browsing of Visualization Recommendations , 2016, IEEE Transactions on Visualization and Computer Graphics.

[32]  Juliana Freire,et al.  Data Polygamy: The Many-Many Relationships among Urban Spatio-Temporal Data Sets , 2016, SIGMOD Conference.

[33]  Trevor Hastie,et al.  Computer Age Statistical Inference: Algorithms, Evidence, and Data Science , 2016 .

[34]  David H. Laidlaw,et al.  A Case Study Using Visualization Interaction Logs and Insight Metrics to Understand How Analysts Arrive at Insights , 2016, IEEE Transactions on Visualization and Computer Graphics.

[35]  Pierre Dragicevic,et al.  The Attraction Effect in Information Visualization , 2017, IEEE Transactions on Visualization and Computer Graphics.

[36]  Tim Kraska,et al.  How Progressive Visualizations Affect Exploratory Analysis , 2017, IEEE Transactions on Visualization and Computer Graphics.

[37]  安藤 寛,et al.  Cross-Validation , 1952, Encyclopedia of Machine Learning and Data Mining.

[38]  P. Pirolli,et al.  The Sensemaking Process and Leverage Points for Analyst Technology as Identified Through Cognitive Task Analysis , 2007 .