Challenges in Evaluating Interactive Visual Machine Learning Systems

In interactive visual machine learning (IVML), humans and machine learning algorithms collaborate to achieve tasks mediated by interactive visual interfaces. This human-in-the-loop approach to machine learning brings forth not only numerous intelligibility, trust, and usability issues, but also many open questions with respect to the evaluation of the IVML system, both as separate components, and as a holistic entity that includes both human and machine intelligence. This article describes the challenges and research gaps identified in an IEEE VIS workshop on the evaluation of IVML systems.

[1]  Christopher Andrews,et al.  The human is the loop: new directions for visual analytics , 2014, Journal of Intelligent Information Systems.

[2]  Jing Yang,et al.  Guidance in the human-machine analytics process , 2018, Vis. Informatics.

[3]  Anastasia Bezerianos,et al.  Evaluation of Interactive Machine Learning Systems , 2018, Human and Machine Learning.

[4]  Jessica Hullman,et al.  Human Factors in Model Interpretability: Industry Practices, Challenges, and Needs , 2020, Proc. ACM Hum. Comput. Interact..

[5]  Minsuk Kahng.  How Does Visualization Help People Learn Deep Learning? Evaluation of GAN Lab , 2019 .

[6]  Alberto Paolo Tonda,et al.  LIDeOGraM: An Interactive Evolutionary Modelling Tool , 2017, Artificial Evolution.

[7]  R. Chang,et al.  Inferential Tasks as a Data-Rich Evaluation Method for Visualization , 2019 .

[8]  Eric Horvitz,et al.  Beyond Accuracy: The Role of Mental Models in Human-AI Team Performance , 2019, HCOMP.

[10]  Alex Endert,et al.  The State of the Art in Integrating Machine Learning into Visual Analytics , 2017, Comput. Graph. Forum.

[11]  Carla E. Brodley,et al.  Dis-function: Learning distance functions interactively , 2012, 2012 IEEE Conference on Visual Analytics Science and Technology (VAST).

[12]  Daniel A. Keim,et al.  Human-centered machine learning through interactive visualization , 2016 .

[13]  Ioana M. Boier-Martin,et al.  Visualization Viewpoints , 2000 .

[14]  Matthew Kay,et al.  In Pursuit of Error: A Survey of Uncertainty Visualization Evaluation , 2019, IEEE Transactions on Visualization and Computer Graphics.

[15]  Pierre Dragicevic,et al.  A Task-Based Taxonomy of Cognitive Biases for Information Visualization , 2020, IEEE Transactions on Visualization and Computer Graphics.

[16]  E. Brink,et al.  Constructing grounded theory : A practical guide through qualitative analysis , 2006 .

[17]  Anastasia Bezerianos,et al.  Evolutionary Visual Exploration: Evaluation of an IEC Framework for Guided Visual Search , 2017, Evolutionary Computation.

[18]  Chris North,et al.  Evaluating Semantic Interaction on Word Embeddings via Simulation , 2020, ArXiv.

[19]  Alex Endert,et al.  Warning, Bias May Occur: A Proposed Approach to Detecting Cognitive Bias in Interactive Visual Analytics , 2017, 2017 IEEE Conference on Visual Analytics Science and Technology (VAST).

[20]  Steven M. Drucker,et al.  Helping Users Sort Faster with Adaptive Machine Learning Recommendations , 2011, INTERACT.

[21]  Sean A. Munson,et al.  Uncertainty Displays Using Quantile Dotplots or CDFs Improve Transit Decision-Making , 2018, CHI.

[22]  Denis Lalanne,et al.  Surveying the complementary role of automatic data analysis and visualization in knowledge discovery , 2009, VAKD '09.

[23]  Wendy E. Mackay,et al.  Users and customizable software : a co-adaptive phenomenon , 1990 .

[24]  Yea-Seul Kim,et al.  A Bayesian Cognition Approach to Improve Data Visualization , 2019, CHI.

[25]  Paul N. Bennett,et al.  Guidelines for Human-AI Interaction , 2019, CHI.

[26]  Daniel A. Keim,et al.  Progressive Learning of Topic Modeling Parameters: A Visual Analytics Framework , 2018, IEEE Transactions on Visualization and Computer Graphics.

[27]  André Calero Valdez,et al.  Priming and Anchoring Effects in Visualization , 2018, IEEE Transactions on Visualization and Computer Graphics.

[28]  Tobias Schreck,et al.  mVis in the Wild: Pre-Study of an Interactive Visual Machine Learning System for Labelling , 2019 .

[29]  Shixia Liu,et al.  Recent research advances on interactive machine learning , 2018, J. Vis..

[30]  Katharina Reinecke,et al.  Explaining the Gap: Visualizing One's Predictions Improves Recall and Comprehension of Data , 2017, CHI.

[31]  Lee Lacy,et al.  Defense Advanced Research Projects Agency (DARPA) Agent Markup Language Computer Aided Knowledge Acquisition , 2005 .

[32]  Samuel J. Gershman,et al.  Human Evaluation of Models Built for Interpretability , 2019, HCOMP.

[33]  Giuseppe Santucci,et al.  The Human User in Progressive Visual Analytics , 2019, EuroVis.

[34]  Eric Horvitz,et al.  Addressing bias in machine learning algorithms: A pilot study on emotion recognition for intelligent systems , 2017, 2017 IEEE Workshop on Advanced Robotics and its Social Impacts (ARSO).