A Review of User Interface Design for Interactive Machine Learning

Interactive Machine Learning (IML) seeks to complement human perception and intelligence by tightly integrating these strengths with the computational power and speed of computers. The interactive process is designed to involve input from the user but does not require the background knowledge or experience that might be necessary to work with more traditional machine learning techniques. Under the IML process, non-experts can apply their domain knowledge and insight over otherwise unwieldy datasets to find patterns of interest or develop complex data-driven applications. This process is co-adaptive in nature and relies on careful management of the interaction between human and machine. User interface design is fundamental to the success of this approach, yet there is a lack of consolidated principles on how such an interface should be implemented. This article presents a detailed review and characterisation of Interactive Machine Learning from an interactive systems perspective. We propose and describe a structural and behavioural model of a generalised IML system and identify solution principles for building effective interfaces for IML. Where possible, these emergent solution principles are contextualised by reference to the broader human-computer interaction literature. Finally, we identify strands of user interface research key to unlocking more efficient and productive non-expert interactive machine learning applications.

[1]  Andrea Kleinsmith,et al.  Embodied Design of Dance Visualisations , 2014, MOCO '14.

[2]  Alex Groce,et al.  You Are the Only Possible Oracle: Effective Test Selection for End Users of Interactive Machine Learning Systems , 2014, IEEE Transactions on Software Engineering.

[3]  Maya Cakmak,et al.  Eliciting good teaching from humans for machine learners , 2014, Artif. Intell..

[4]  Zoubin Ghahramani,et al.  Probabilistic machine learning and artificial intelligence , 2015, Nature.

[5]  Jeffrey M. Bradshaw,et al.  Trust in Automation , 2013, IEEE Intelligent Systems.

[6]  Michael S. Bernstein,et al.  Flock: Hybrid Crowd-Machine Learning Classifiers , 2015, CSCW.

[7]  Desney S. Tan,et al.  CueFlik: interactive concept learning in image search , 2008, CHI.

[8]  Ben Shneiderman,et al.  The eyes have it: a task by data type taxonomy for information visualizations , 1996, Proceedings 1996 IEEE Symposium on Visual Languages.

[9]  Hema Raghavan,et al.  Active Learning with Feedback on Features and Instances , 2006, J. Mach. Learn. Res..

[10]  Miriam A. M. Capretz,et al.  MLaaS: Machine Learning as a Service , 2015, 2015 IEEE 14th International Conference on Machine Learning and Applications (ICMLA).

[11]  Atau Tanaka,et al.  Machine Learning of Personal Gesture Variation in Music Conducting , 2016, CHI.

[12]  Chris North,et al.  Bridging the gap between user intention and model parameters for human-in-the-loop data analytics , 2016, HILDA '16.

[13]  Chris North,et al.  Semantic interaction for visual text analytics , 2012, CHI.

[14]  Camelia-Mihaela Pintea,et al.  Towards interactive Machine Learning (iML): Applying Ant Colony Algorithms to Solve the Traveling Salesman Problem with the Human-in-the-Loop Approach , 2016, CD-ARES.

[15]  B. Argall,et al.  Human-in-the-Loop Optimization of Shared Autonomy in Assistive Robotics , 2017, IEEE Robotics and Automation Letters.

[16]  Desney S. Tan,et al.  Effective End-User Interaction with Machine Learning , 2011, AAAI.

[17]  Michel Beaudouin-Lafon,et al.  Designing interaction, not interfaces , 2004, AVI.

[18]  Weng-Keen Wong,et al.  End-user feature labeling: a locally-weighted regression approach , 2011, IUI '11.

[19]  Anind K. Dey,et al.  a CAPpella: programming by demonstration of context-aware applications , 2004, CHI.

[20]  David Maxwell Chickering,et al.  ModelTracker: Redesigning Performance Analysis Tools for Machine Learning , 2015, CHI.

[21]  Ben Shneiderman,et al.  Readings in information visualization - using vision to think , 1999 .

[22]  Burr Settles,et al.  Active Learning Literature Survey , 2009 .

[23]  Peng Dai,et al.  AppGrouper: Knowledge-based Interactive Clustering Tool for App Search Results , 2016, IUI.

[24]  Andreas Holzinger,et al.  A Domain-Expert Centered Process Model for Knowledge Discovery in Medical Research: Putting the Expert-in-the-Loop , 2015, BIH.

[25]  Andrea Kleinsmith,et al.  Embodied design of full bodied interaction with virtual humans , 2015, MOCO.

[26]  Neal R. Harvey,et al.  Interactive image quantification tools in nuclear material forensics , 2011, Electronic Imaging.

[27]  Thomas G. Dietterich,et al.  Interacting meaningfully with machine learning systems: Three experiments , 2009, Int. J. Hum. Comput. Stud..

[28]  Carla E. Brodley,et al.  Deploying an interactive machine learning system in an evidence-based practice center: abstrackr , 2012, IHI '12.

[29]  Per Ola Kristensson,et al.  On the benefits of confidence visualization in speech recognition , 2008, CHI.

[30]  Ratul Mahajan,et al.  Human-Guided Machine Learning for Fast and Accurate Network Alarm Triage , 2011, IJCAI.

[31]  Paolo Lombardi,et al.  Filtering Surveillance Image Streams by Interactive Machine Learning , 2011, Multimedia Analysis, Processing and Communications.

[32]  Ben Shneiderman,et al.  The future of interactive systems and the emergence of direct manipulation , 1982 .

[33]  Nicolas Gaud,et al.  A Review and Taxonomy of Interactive Optimization Methods in Operations Research , 2015, ACM Trans. Interact. Intell. Syst..

[34]  Gautham J. Mysore,et al.  ISSE: an interactive source separation editor , 2014, CHI.

[35]  Mark A. Girolami,et al.  Putting the Scientist in the Loop -- Accelerating Scientific Progress with Interactive Machine Learning , 2014, 2014 22nd International Conference on Pattern Recognition.

[36]  Gaël Varoquaux,et al.  Scikit-learn: Machine Learning in Python , 2011, J. Mach. Learn. Res..

[37]  Jakob Nielsen,et al.  Heuristic Evaluation of Prototypes (individual) , 2022 .

[38]  Stephanie Rosenthal,et al.  Towards maximizing the accuracy of human-labeled sensor data , 2010, IUI '10.

[39]  Stuart K. Card,et al.  Information foraging in information access environments , 1995, CHI '95.

[40]  Joshua B. Tenenbaum,et al.  Automatic Construction and Natural-Language Description of Nonparametric Regression Models , 2014, AAAI.

[41]  Carlos Guestrin,et al.  "Why Should I Trust You?": Explaining the Predictions of Any Classifier , 2016, ArXiv.

[42]  James A. Landay,et al.  Investigating statistical machine learning as a tool for software development , 2008, CHI.

[43]  Perry R. Cook,et al.  Real-time human interaction with supervised learning algorithms for music composition and performance , 2011 .

[44]  Advait Sarkar,et al.  Confidence, command, complexity: metamodels for structured interaction with machine intelligence , 2015, PPIG.

[45]  Heiko Wersing,et al.  Interactive online learning for obstacle classification on a mobile robot , 2015, 2015 International Joint Conference on Neural Networks (IJCNN).

[46]  Thecla Schiphorst,et al.  GaussBox: Prototyping Movement Interaction with Interactive Visualizations of Machine Learning , 2016, CHI Extended Abstracts.

[47]  Yang Li,et al.  Gesture script: recognizing gestures and their structure using rendering scripts and interactively trained parts , 2014, CHI.

[48]  Kayur Patel,et al.  Scalable and Interpretable Data Representation for High-Dimensional, Complex Data , 2015, AAAI.

[49]  Kevin D. Ashley,et al.  Applying an Interactive Machine Learning Approach to Statutory Analysis , 2015, JURIX.

[50]  Chris North,et al.  Information Visualization , 2008, Lecture Notes in Computer Science.

[51]  Ian H. Witten,et al.  Interactive machine learning: letting users build classifiers , 2002, Int. J. Hum. Comput. Stud..

[52]  Rayid Ghani,et al.  Interactive learning for efficiently detecting errors in insurance claims , 2011, KDD.

[53]  M. Sheelagh T. Carpendale,et al.  Evaluating Information Visualizations , 2008, Information Visualization.

[54]  Bertrand Rivet,et al.  Adding Human Learning in Brain--Computer Interfaces (BCIs) , 2015, ACM Trans. Comput. Hum. Interact..

[55]  Andrew Howes,et al.  Adaptive Interaction: A Utility Maximization Approach to Understanding Human Interaction with Technology , 2013, Adaptive Interaction: A Utility Maximization Approach to Understanding Human Interaction with Technology.

[56]  Stephen E. Fienberg,et al.  Test time feature ordering with FOCUS: interactive predictions with minimal user burden , 2016, UbiComp.

[57]  Peter Stone,et al.  Framing reinforcement learning from human reward: Reward positivity, temporal discounting, episodicity, and performance , 2015, Artif. Intell..

[58]  Siddhartha S. Srinivasa,et al.  Minimizing user cost for shared autonomy , 2016, 2016 11th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[59]  Perry R. Cook,et al.  Human model evaluation in interactive supervised learning , 2011, CHI.

[60]  John Riedl,et al.  An operator interaction framework for visualization systems , 1998, Proceedings IEEE Symposium on Information Visualization (Cat. No.98TB100258).

[61]  Eric Horvitz,et al.  Principles of mixed-initiative user interfaces , 1999, CHI '99.

[62]  Wendy E. Mackay,et al.  Musink: composing music through augmented drawing , 2009, CHI.

[63]  Pierre Dragicevic,et al.  Assessing the Effect of Visualizations on Bayesian Reasoning through Crowdsourcing , 2012, IEEE Transactions on Visualization and Computer Graphics.

[64]  Carla E. Brodley,et al.  Dis-function: Learning distance functions interactively , 2012, 2012 IEEE Conference on Visual Analytics Science and Technology (VAST).

[65]  Christian Biemann,et al.  Interactive and Iterative Annotation for Biomedical Entity Recognition , 2015, BIH.

[66]  Fred A. Hamprecht,et al.  Automated Detection and Segmentation of Synaptic Contacts in Nearly Isotropic Serial Electron Microscopy Images , 2011, PloS one.

[67]  Desney S. Tan,et al.  CueTIP: a mixed-initiative interface for correcting handwriting errors , 2006, UIST.

[68]  Qiang Sun,et al.  Explanation-Augmented SVM: an approach to incorporating domain knowledge into SVM learning , 2005, ICML.

[69]  Kristin Branson,et al.  JAABA: interactive machine learning for automatic annotation of animal behavior , 2013, Nature Methods.

[70]  James Fogarty,et al.  BeatBox: end-user interactive definition and training of recognizers for percussive vocalizations , 2014, AVI.

[71]  Daniel A. Keim,et al.  The Role of Uncertainty, Awareness, and Trust in Visual Analytics , 2016, IEEE Transactions on Visualization and Computer Graphics.

[72]  Been Kim,et al.  iBCM: Interactive Bayesian Case Model Empowering Humans via Intuitive Interaction , 2015 .

[73]  E. C. Chua,et al.  Improved patient specific seizure detection during pre-surgical evaluation , 2011, Clinical Neurophysiology.

[74]  Ratul Mahajan,et al.  CueT: human-guided fast and accurate network alarm triage , 2011, CHI.

[75]  Scott R. Klemmer,et al.  Authoring sensor-based interactions by demonstration with direct manipulation and pattern recognition , 2007, CHI.

[76]  Andreas Holzinger,et al.  Human-Computer Interaction and Knowledge Discovery (HCI-KDD): What Is the Benefit of Bringing Those Two Fields to Work Together? , 2013, CD-ARES.

[77]  Alan F. Blackwell,et al.  Interactive visual machine learning in spreadsheets , 2015, 2015 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC).

[78]  Andrea Kleinsmith,et al.  Applying the CASSM Framework to Improving End User Debugging of Interactive Machine Learning , 2015, IUI.

[79]  Neal R. Harvey,et al.  User-driven sampling strategies in image exploitation , 2013, Electronic Imaging.

[80]  Andrea Kleinsmith,et al.  Customizing by doing for responsive video game characters , 2013, Int. J. Hum. Comput. Stud..

[81]  Desney S. Tan,et al.  EnsembleMatrix: interactive visualization to support machine learning with multiple classifiers , 2009, CHI.

[82]  Desney S. Tan,et al.  Interactive optimization for steering machine classification , 2010, CHI.

[83]  B. J. Fogg,et al.  The elements of computer credibility , 1999, CHI '99.

[84]  Rebecca Fiebrink,et al.  Using Interactive Machine Learning to Support Interface Development Through Workshops with Disabled People , 2015, CHI.

[85]  Sumit Basu,et al.  Learning to generalize for complex selection tasks , 2009, IUI.

[86]  Don R. Hush,et al.  Interactive Machine Learning in Data Exploitation , 2013, Computing in Science & Engineering.

[87]  Wai-Tat Fu,et al.  Leveraging the crowd to improve feature-sentiment analysis of user reviews , 2013, IUI '13.

[88]  Todd Kulesza,et al.  Structured labeling for facilitating concept evolution in machine learning , 2014, CHI.

[89]  Fujio Tsutsumi,et al.  A Method to Recognize and Count Leaves on the Surface of a River Using User's Knowledge about Color of Leaves , 2009, PAKDD Workshops.

[90]  Peter Kontschieder,et al.  Setwise Comparison: Consistent, Scalable, Continuum Labels for Computer Vision , 2016, CHI.

[91]  Geoffrey E. Hinton,et al.  Visualizing Data using t-SNE , 2008 .

[92]  James Fogarty,et al.  Regroup: interactive machine learning for on-demand group creation in social networks , 2012, CHI.

[93]  Igor Jurisica,et al.  Knowledge Discovery and Data Mining in Biomedical Informatics: The Future Is in Integrative, Interactive Machine Learning Solutions , 2014, Interactive Knowledge Discovery and Data Mining in Biomedical Informatics.

[94]  Matjaz Gams,et al.  Combining human analysis and machine data mining to obtain credible data relations , 2014, Inf. Sci..

[95]  Alfons Juan-Císcar,et al.  A prototype for interactive speech transcription balancing error and supervision effort , 2012, IUI '12.

[96]  Desney S. Tan,et al.  Examining multiple potential models in end-user interactive concept learning , 2010, CHI.

[97]  Maya Cakmak,et al.  Designing Interactions for Robot Active Learners , 2010, IEEE Transactions on Autonomous Mental Development.

[98]  Wendy E. Mackay,et al.  Human-Centred Machine Learning , 2016, CHI Extended Abstracts.

[99]  Weng-Keen Wong,et al.  Too much, too little, or just right? Ways explanations impact end users' mental models , 2013, 2013 IEEE Symposium on Visual Languages and Human Centric Computing.

[100]  Saleema Amershi,et al.  Designing for effective end-user interaction with machine learning , 2011, UIST '11 Adjunct.

[101]  N. Moray,et al.  Trust in automation. Part II. Experimental studies of trust and human intervention in a process control simulation. , 1996, Ergonomics.

[102]  Ashish Kapoor,et al.  FeatureInsight: Visual support for error-driven feature ideation in text classification , 2015, 2015 IEEE Conference on Visual Analytics Science and Technology (VAST).

[103]  David Gotz,et al.  Progressive Visual Analytics: User-Driven Visual Exploration of In-Progress Analytics , 2014, IEEE Transactions on Visualization and Computer Graphics.

[104]  Maya Cakmak,et al.  Power to the People: The Role of Humans in Interactive Machine Learning , 2014, AI Mag..

[105]  Weng-Keen Wong,et al.  Principles of Explanatory Debugging to Personalize Interactive Machine Learning , 2015, IUI.

[106]  Weng-Keen Wong,et al.  Towards recognizing "cool": can end users help computer vision recognize subjective attributes of objects in images? , 2012, IUI '12.

[107]  Donald A. Norman,et al.  Some observations on mental models , 1987 .

[108]  Kemal Kilic,et al.  An interactive machine-learning-based electronic fraud and abuse detection system in healthcare insurance , 2015, Appl. Soft Comput..

[109]  Jerry Alan Fails,et al.  Interactive machine learning , 2003, IUI '03.

[110]  Ian H. Witten,et al.  The WEKA data mining software: an update , 2009, SKDD.

[111]  Jeff Hemsley,et al.  Mixed-initiative social media analytics at the World Bank: Observations of citizen sentiment in Twitter data to explore "trust" of political actors and state institutions and its relationship to social protest , 2015, 2015 IEEE International Conference on Big Data (Big Data).

[112]  Pengcheng Shi,et al.  An Expert-in-the-loop Paradigm for Learning Medical Image Grouping , 2016, PAKDD.

[113]  Alex T. Pang,et al.  Approaches to uncertainty visualization , 1996, The Visual Computer.