Evaluation of Interactive Machine Learning Systems

The evaluation of interactive machine learning systems remains a difficult task. These systems learn from and adapt to the human, but at the same time, the human receives feedback and adapts to the system. Getting a clear understanding of these subtle mechanisms of co-operation and co-adaptation is challenging. In this chapter, we report on our experience in designing and evaluating various interactive machine learning applications from different domains. We argue for coupling two types of validation: algorithm-centered analysis, to study the computational behaviour of the system; and human-centered evaluation, to observe the utility and effectiveness of the application for end-users. We use a visual analytics application for guided search, built using an interactive evolutionary approach, as an exemplar of our work. Our observation is that human-centered design and evaluation complement algorithmic analysis, and can play an important role in addressing the "black-box" effect of machine learning. Finally, we discuss research opportunities that require human-computer interaction methodologies, in order to support both the visible and hidden roles that humans play in interactive machine learning.

[1]  Jacob O. Wobbrock,et al.  Understanding usability practices in complex domains , 2010, CHI.

[2]  Jürgen Branke,et al.  Evolutionary optimization in uncertain environments-a survey , 2005, IEEE Transactions on Evolutionary Computation.

[3]  Maya Cakmak,et al.  Power to the People: The Role of Humans in Interactive Machine Learning , 2014, AI Mag..

[4]  Shumin Zhai.  Editorial: TOCHI turns twenty , 2014, TCHI.

[5]  Risto Miikkulainen,et al.  Evolving Neural Networks through Augmenting Topologies , 2002, Evolutionary Computation.

[6]  James R. Eagan,et al.  How Data Workers Cope with Uncertainty: A Task Characterisation Study , 2017, CHI.

[7]  Jaime Teevan,et al.  Implicit feedback for inferring user preference: a bibliography , 2003, SIGF.

[8]  Georges G. Grinstein.  Harnessing the Human in Knowledge Discovery , 1996, KDD.

[9]  Pierrick Legrand,et al.  Interactive evolution for cochlear implants fitting , 2007, Genetic Programming and Evolvable Machines.

[10]  R. Grossman,et al.  Graph-theoretic scagnostics , 2005, IEEE Symposium on Information Visualization, 2005. INFOVIS 2005..

[11]  Chris North,et al.  Semantic Interaction for Sensemaking: Inferring Analytical Reasoning for Model Steering , 2012, IEEE Transactions on Visualization and Computer Graphics.

[12]  Hideyuki Takagi,et al.  Interactive Evolutionary Computation : System Optimization Based on Human Subjective Evaluation , 1998 .

[13]  Norman W. Paton,et al.  Observing the Data Scientist: Using Manual Corrections As Implicit Feedback , 2017, HILDA@SIGMOD.

[14]  Evelyne Lutton,et al.  Balancing User Interaction and Control in Bayesian Network Structure Learning , 2013 .

[15]  Anastasia Bezerianos,et al.  Evolutionary Visual Exploration: Evaluation of an IEC Framework for Guided Visual Search , 2017, Evolutionary Computation.

[16]  Celine Latulipe,et al.  Quantifying the Creativity Support of Digital Tools through the Creativity Support Index , 2014, ACM Trans. Comput. Hum. Interact..

[17]  Jingrui He,et al.  RCLens: Interactive Rare Category Exploration and Identification , 2018, IEEE Transactions on Visualization and Computer Graphics.

[18]  Anastasia Bezerianos,et al.  Evolutionary visual exploration: experimental analysis of algorithm behaviour , 2013, GECCO.

[19]  Christopher G. Healey,et al.  Interest Driven Navigation in Visualization , 2012, IEEE Transactions on Visualization and Computer Graphics.

[20]  Riccardo Poli,et al.  Genetic Programming with User-Driven Selection : Experiments on the Evolution of Algorithms for Image Enhancement , 1997 .

[21]  Alberto Paolo Tonda,et al.  Balancing User Interaction and Control in BNSL , 2013, Artificial Evolution.

[22]  Jeffrey Horn,et al.  Handbook of evolutionary computation , 1997 .

[23]  Gautham J. Mysore,et al.  ISSE: an interactive source separation editor , 2014, CHI.

[24]  Jaegul Choo,et al.  UTOPIAN: User-Driven Topic Modeling Based on Interactive Nonnegative Matrix Factorization , 2013, IEEE Transactions on Visualization and Computer Graphics.

[25]  David E. Goldberg,et al.  Genetic Algorithms in Search Optimization and Machine Learning , 1988 .

[26]  Télécom ParisTech,et al.  Research Prospects in the Design and Evaluation of Interactive Evolutionary Systems for Art and Science , 2016 .

[27]  Paul L. Rosin,et al.  Abstract Art by Shape Classification , 2013, IEEE Transactions on Visualization and Computer Graphics.

[28]  Takeo Igarashi,et al.  SelPh: Progressive Learning and Support of Manual Photo Color Enhancement , 2016, CHI.

[29]  W. Mackay.  Responding to cognitive overload : Co-adaptation between users and technology , 2000 .

[30]  M. Sheelagh T. Carpendale,et al.  Evaluating Information Visualizations , 2008, Information Visualization.

[31]  Chris North,et al.  An insight-based methodology for evaluating bioinformatics visualizations , 2005, IEEE Transactions on Visualization and Computer Graphics.

[32]  Carla E. Brodley,et al.  Dis-function: Learning distance functions interactively , 2012, 2012 IEEE Conference on Visual Analytics Science and Technology (VAST).

[33]  T. Munzner,et al.  Dimensionality Reduction in the Wild : Gaps and Guidance , 2012 .

[34]  Jesus J. Caban,et al.  A Grammar-based Approach for Modeling User Interactions and Generating Suggestions During the Data Exploration Process , 2017, IEEE Transactions on Visualization and Computer Graphics.

[35]  Min Chen,et al.  Transformation of an Uncertain Video Search Pipeline to a Sketch-Based Visual Analytics Loop , 2013, IEEE Transactions on Visualization and Computer Graphics.

[36]  Eric Horvitz,et al.  Principles of mixed-initiative user interfaces , 1999, CHI '99.

[37]  Chris North,et al.  Observation-Level Interaction with Clustering and Dimension Reduction Algorithms , 2017, HILDA@SIGMOD.

[38]  Andreas Holzinger,et al.  Interactive machine learning for health informatics: when do we need the human-in-the-loop? , 2016, Brain Informatics.

[39]  Evelyne Lutton,et al.  EvoGraphDice: Interactive evolution for visual analytics , 2012, 2012 IEEE Congress on Evolutionary Computation.

[40]  Evelyne Lutton,et al.  Evolution of Fractal Shapes for Artists and Designers , 2006, Int. J. Artif. Intell. Tools.

[41]  Lin Gao,et al.  Active Exploration of Large 3D Model Repositories , 2015, IEEE Transactions on Visualization and Computer Graphics.

[42]  Anastasia Bezerianos,et al.  A Mixed Approach for the Evaluation of a Guided Exploratory Visualization System , 2015, EuroRV³@EuroVis.

[43]  Yann Landrin-Schweitzer,et al.  Introducing lateral thinking in search engines , 2006, Genetic Programming and Evolvable Machines.

[44]  James Fogarty,et al.  Regroup: interactive machine learning for on-demand group creation in social networks , 2012, CHI.

[45]  Fatih Korkmaz,et al.  Feedback-driven interactive exploration of large multidimensional data supported by visual classifier , 2014, 2014 IEEE Conference on Visual Analytics Science and Technology (VAST).

[46]  Christopher Ré,et al.  Data programming with DDLite: putting humans in a different part of the loop , 2016, HILDA '16.

[47]  Chris North,et al.  Semantic interaction for visual text analytics , 2012, CHI.

[48]  Anastasia Bezerianos,et al.  Evolutionary Visual Exploration: Evaluation With Expert Users , 2013, Comput. Graph. Forum.

[49]  Thomas Ertl,et al.  Visual Classifier Training for Text Document Retrieval , 2012, IEEE Transactions on Visualization and Computer Graphics.

[50]  Jean-Daniel Fekete,et al.  Interactive Random Graph Generation with Evolutionary Algorithms , 2012, Graph Drawing.

[51]  Daniel A. Keim,et al.  Human-centered machine learning through interactive visualization , 2016 .

[52]  Amedeo Cesta,et al.  Evaluating Mixed-Initiative Systems: An Experimental Approach , 2006, ICAPS.

[53]  Zbigniew Michalewicz,et al.  Handbook of Evolutionary Computation , 1997 .