Learning and Anticipating Future Actions During Exploratory Data Analysis

The goal of visual analytics is to create a symbiosis between human and computer by leveraging their unique strengths. While this model has demonstrated immense success, we are yet to realize the full potential of such a human-computer partnership. In a perfect collaborative mixed-initiative system, the computer must possess skills for learning and anticipating the users' needs. Addressing this gap, we propose a framework for inferring focus areas from passive observations of the user's actions, thereby allowing accurate predictions of future events. We evaluate this technique with a crime map and demonstrate that users' clicks appear in our prediction set 95% - 97% of the time. Further analysis shows that we can achieve high prediction accuracy typically after three clicks. Altogether, we show that passive observations of interaction data can reveal valuable information that will allow the system to learn and anticipate future events, laying the foundation for next-generation tools.

[1]  C. Koch,et al.  A saliency-based search mechanism for overt and covert shifts of visual attention , 2000, Vision Research.

[2]  A. Joshi,et al.  Web mining: research and practice , 2004, Computing in Science & Engineering.

[3]  Daniel Afergan Using Passive Input to Adapt Visualization Systems to the Individual , 2013 .

[4]  Eric Horvitz,et al.  Uncertainty, Action, and Interaction: In Pursuit of Mixed-Initiative Computing , 2016 .

[5]  William Ribarsky,et al.  Visual analytics for complex concepts using a human cognition model , 2008, 2008 IEEE Symposium on Visual Analytics Science and Technology.

[6]  Michelle X. Zhou,et al.  Interactive Visual Synthesis of Analytic Knowledge , 2006, 2006 IEEE Symposium On Visual Analytics Science And Technology.

[7]  Alvitta Ottley,et al.  Balancing Human and Machine Contributions in Human Computation Systems , 2013, Handbook of Human Computation.

[8]  Ken Brodlie,et al.  GRASPARC-A problem solving environment integrating computation and visualization , 1993, Proceedings Visualization '93.

[9]  Jean Scholtz,et al.  Glass Box: An Instrumented Infrastructure for Supporting Human Interaction with Information , 2005, Proceedings of the 38th Annual Hawaii International Conference on System Sciences.

[10]  Tovi Grossman,et al.  The bubble cursor: enhancing target acquisition by dynamic resizing of the cursor's activation area , 2005, CHI.

[11]  Ben Shneiderman,et al.  Readings in information visualization - using vision to think , 1999 .

[12]  Pat Hanrahan,et al.  Enhancing Visual Analysis of Network Traffic Using a Knowledge Representation , 2006, 2006 IEEE Symposium On Visual Analytics Science And Technology.

[13]  Roman Garnett,et al.  Predicting unexpected influxes of players in EVE online , 2014, 2014 IEEE Conference on Computational Intelligence and Games.

[14]  I. V. Ramakrishnan,et al.  Model-driven Visual Analytics , 2008, 2008 IEEE Symposium on Visual Analytics Science and Technology.

[15]  Jeffrey Heer,et al.  Graphical Histories for Visualization: Supporting Analysis, Communication, and Evaluation , 2008, IEEE Transactions on Visualization and Computer Graphics.

[16]  Krzysztof Z. Gajos,et al.  SUPPLE: automatically generating user interfaces , 2004, IUI '04.

[17]  Eric Horvitz,et al.  Models of attention in computing and communication , 2003, Commun. ACM.

[18]  C. Koch,et al.  Computational modelling of visual attention , 2001, Nature Reviews Neuroscience.

[19]  Chris North,et al.  Semantic interaction for visual text analytics , 2012, CHI.

[20]  Chris North,et al.  Semantic Interaction for Sensemaking: Inferring Analytical Reasoning for Model Steering , 2012, IEEE Transactions on Visualization and Computer Graphics.

[21]  John T. Stasko,et al.  Low-level components of analytic activity in information visualization , 2005, IEEE Symposium on Information Visualization, 2005. INFOVIS 2005..

[22]  Sarah Stein,et al.  METROPOLITAN POLICE DEPARTMENT, WASHINGTON, D.C. Major Case/Cold Case Squads Standard Operating Procedures , 2010 .

[23]  Daniel A. Keim,et al.  Information Visualization and Visual Data Mining , 2002, IEEE Trans. Vis. Comput. Graph..

[24]  Pattie Maes,et al.  Footprints: history-rich tools for information foraging , 1999, CHI '99.

[25]  N. Gordon,et al.  Novel approach to nonlinear/non-Gaussian Bayesian state estimation , 1993 .

[26]  Hendrik Blockeel,et al.  Web mining research: a survey , 2000, SKDD.

[27]  Cláudio T. Silva,et al.  Managing Rapidly-Evolving Scientific Workflows , 2006, IPAW.

[28]  Edward Cutrell,et al.  An eye tracking study of the effect of target rank on web search , 2007, CHI.

[29]  David H. Laidlaw,et al.  A Case Study Using Visualization Interaction Logs and Insight Metrics to Understand How Analysts Arrive at Insights , 2016, IEEE Transactions on Visualization and Computer Graphics.

[30]  Nadir Weibel,et al.  Embedded Merge & Split: Visual Adjustment of Data Grouping , 2019, IEEE Transactions on Visualization and Computer Graphics.

[31]  William Ribarsky,et al.  Recovering Reasoning Processes from User Interactions , 2009, IEEE Computer Graphics and Applications.

[32]  Alex Endert,et al.  Finding Waldo: Learning about Users from their Interactions , 2014, IEEE Transactions on Visualization and Computer Graphics.

[33]  Huahai Yang,et al.  Personality as a Predictor of User Strategy: How Locus of Control Affects Search Strategies on Tree Visualizations , 2015, CHI.

[34]  David S. Ebert,et al.  Volume composition and evaluation using eye-tracking data , 2010, TAP.

[35]  Carla E. Brodley,et al.  Dis-function: Learning distance functions interactively , 2012, 2012 IEEE Conference on Visual Analytics Science and Technology (VAST).

[36]  Christof Koch,et al.  A Model of Saliency-Based Visual Attention for Rapid Scene Analysis , 2009 .

[37]  Steven F. Roth,et al.  On the semantics of interactive visualizations , 1996, Proceedings IEEE Symposium on Information Visualization '96.

[38]  John T. Stasko,et al.  Toward a Deeper Understanding of the Role of Interaction in Information Visualization , 2007, IEEE Transactions on Visualization and Computer Graphics.

[39]  Michelle X. Zhou,et al.  Characterizing users’ visual analytic activity for insight provenance , 2008, 2008 IEEE Symposium on Visual Analytics Science and Technology.

[40]  S Ullman,et al.  Shifts in selective visual attention: towards the underlying neural circuitry. , 1985, Human neurobiology.

[41]  Daniel A. Keim,et al.  Visual Analytics: Definition, Process, and Challenges , 2008, Information Visualization.

[42]  Carl Gutwin,et al.  Target assistance for subtly balancing competitive play , 2011, CHI.

[43]  Mark Bailey,et al.  The Grammar of Graphics , 2007, Technometrics.

[44]  Cláudio T. Silva,et al.  Provenance for Computational Tasks: A Survey , 2008, Computing in Science & Engineering.

[45]  Simon J. Godsill,et al.  On sequential Monte Carlo sampling methods for Bayesian filtering , 2000, Stat. Comput..

[46]  Jean-Daniel Fekete,et al.  Task taxonomy for graph visualization , 2006, BELIV '06.

[47]  Jaideep Srivastava,et al.  Web usage mining: discovery and applications of usage patterns from Web data , 2000, SKDD.

[48]  Daniel A. Keim,et al.  Mastering the Information Age - Solving Problems with Visual Analytics , 2010 .

[49]  Jacques Bertin,et al.  Semiology of Graphics - Diagrams, Networks, Maps , 2010 .

[50]  Alan J. Dix,et al.  Starting simple: adding value to static visualisation through simple interaction , 1998, AVI '98.

[51]  Evan M. Peck Designing Brain-Computer Interfaces for Intelligent Information Delivery Systems , 2014 .

[52]  Cristina Conati,et al.  Individual user characteristics and information visualization: connecting the dots through eye tracking , 2013, CHI.

[53]  Frederick Jelinek,et al.  Statistical methods for speech recognition , 1997 .

[54]  Steven K. Feiner,et al.  Visual task characterization for automated visual discourse synthesis , 1998, CHI.

[55]  Alex Endert,et al.  Visualization by Demonstration: An Interaction Paradigm for Visual Data Exploration , 2017, IEEE Transactions on Visualization and Computer Graphics.

[56]  Krzysztof Z. Gajos,et al.  Decision-Theoretic User Interface Generation , 2008, AAAI.

[57]  John T. Stasko,et al.  The Science of Interaction , 2009, Inf. Vis..

[58]  Daniel Afergan,et al.  Designing Implicit Interfaces for Physiological Computing , 2015, ACM Trans. Comput. Hum. Interact..

[59]  Michael Stonebraker,et al.  Dynamic Prefetching of Data Tiles for Interactive Visualization , 2016, SIGMOD Conference.

[60]  Lane Harrison,et al.  Patterns and Pace: Quantifying Diverse Exploration Behavior with Visualizations on the Web , 2019, IEEE Transactions on Visualization and Computer Graphics.

[61]  Curry Guinn,et al.  Mixed-initiative interaction , 1999 .

[62]  Ben Shneiderman,et al.  The eyes have it: a task by data type taxonomy for information visualizations , 1996, Proceedings 1996 IEEE Symposium on Visual Languages.

[63]  Cláudio T. Silva,et al.  VisTrails: visualization meets data management , 2006, SIGMOD Conference.

[64]  Stephen Chi-fai Chan,et al.  Your Mouse Reveals Your Next Activity: Towards Predicting User Intention from Mouse Interaction , 2017, 2017 IEEE 41st Annual Computer Software and Applications Conference (COMPSAC).

[65]  H. Theisel,et al.  ExPlates: Spatializing Interactive Analysis to Scaffold Visual Exploration , 2013, Comput. Graph. Forum.

[66]  MAGDALINI EIRINAKI,et al.  Web mining for web personalization , 2003, TOIT.

[67]  Daniel Paurat Interactive Exploration of Larger Pattern Collections : A Case Study on a Cocktail Dataset , 2014 .

[68]  Alex Endert,et al.  Characterizing Provenance in Visualization and Data Analysis: An Organizational Framework of Provenance Types and Purposes , 2016, IEEE Transactions on Visualization and Computer Graphics.

[69]  Erik L. L. Sonnhammer,et al.  A Hidden Markov Model for Predicting Transmembrane Helices in Protein Sequences , 1998, ISMB.

[70]  Krzysztof Z. Gajos,et al.  Improving the performance of motor-impaired users with automatically-generated, ability-based interfaces , 2008, CHI.

[71]  Matthias Scheutz,et al.  Sensing cognitive multitasking for a brain-based adaptive user interface , 2011, CHI.

[72]  Eric Horvitz,et al.  Reflections on Challenges and Promises of Mixed-Initiative Interaction , 2007, AI Mag..

[73]  Cláudio T. Silva,et al.  VisTrails: enabling interactive multiple-view visualizations , 2005, VIS 05. IEEE Visualization, 2005..

[74]  Alex Endert,et al.  Warning, Bias May Occur: A Proposed Approach to Detecting Cognitive Bias in Interactive Visual Analytics , 2017, 2017 IEEE Conference on Visual Analytics Science and Technology (VAST).

[75]  Andreas Buja,et al.  Interactive High-Dimensional Data Visualization , 1996 .

[76]  Peter Pirolli,et al.  Information Foraging , 2009, Encyclopedia of Database Systems.

[77]  Ranjeev Mittu,et al.  Towards a Context-Aware Proactive Decision Support Framework , 2013, STIDS.

[78]  Cristina Conati,et al.  User-adaptive information visualization: using eye gaze data to infer visualization tasks and user cognitive abilities , 2013, IUI '13.

[79]  Chris North,et al.  Analytic provenance: process+interaction+insight , 2011, CHI Extended Abstracts.

[80]  Biing-Hwang Juang,et al.  Fundamentals of speech recognition , 1993, Prentice Hall signal processing series.

[81]  Jesus J. Caban,et al.  A Grammar-based Approach for Modeling User Interactions and Generating Suggestions During the Data Exploration Process , 2017, IEEE Transactions on Visualization and Computer Graphics.

[82]  Isaac Cho,et al.  The Anchoring Effect in Decision-Making with Visual Analytics , 2017, 2017 IEEE Conference on Visual Analytics Science and Technology (VAST).

[83]  Sean R. Eddy,et al.  Biological Sequence Analysis: Probabilistic Models of Proteins and Nucleic Acids , 1998 .

[84]  R. Jordan Crouser,et al.  Online Submission ID: 200 An Affordance-Based Framework for Human Computation and Human-Computer Collaboration , 2022 .

[85]  Eric Horvitz,et al.  Principles of mixed-initiative user interfaces , 1999, CHI '99.

[86]  Daniel A. Keim,et al.  Viewing Visual Analytics as Model Building , 2018, Comput. Graph. Forum.

[87]  Pierre Dragicevic,et al.  The Attraction Effect in Information Visualization , 2017, IEEE Transactions on Visualization and Computer Graphics.

[88]  Jarke J. van Wijk,et al.  Supporting the analytical reasoning process in information visualization , 2008, CHI.