Interactive machine learning for user-innovation toolkits: an action design research approach

Machine learning offers great potential to developers and end users in the creative industries. However, to better support creative software developers' needs and empower them as machine learning users and innovators, the usability of and developer experience with machine learning tools must be considered and better understood. This thesis asks the following research questions: How can we apply a user-centred approach to the design of developer tools for rapid prototyping with Interactive Machine Learning? In what ways can we design better developer tools to accelerate and broaden innovation with machine learning? This thesis presents a three-year longitudinal action research study that I undertook within a multi-institutional consortium leading the EU H2020 -funded Innovation Action RAPID-MIX. The scope of the research presented here was the application of a user-centred approach to the design and evaluation of developer tools for rapid prototyping and product development with machine learning. This thesis presents my work in collaboration with other members of RAPID-MIX, including design and deployment of a user-centred methodology for the project, interventions for gathering requirements with RAPID-MIX consortium stakeholders and end users, and prototyping, development and evaluation of a software development toolkit for interactive machine learning. This thesis contributes with new understanding about the consequences and implications of a user-centred approach to the design and evaluation of developer tools for rapid prototyping of interactive machine learning systems. This includes 1) new understanding about the goals, needs, expectations, and challenges facing creative machine-learning non-expert developers and 2) an evaluation of the usability and design trade-offs of a toolkit for rapid prototyping with interactive machine learning. This thesis also contributes with 3) a methods framework of User-Centred Design Actions for harmonising User-Centred Design with Action Research and supporting the collaboration between action researchers and practitioners working in rapid innovation actions, and 4) recommendations for applying Action Research and User-Centred Design in similar contexts and scale.

[1]  Gilles Louppe,et al.  Independent consultant , 2013 .

[2]  Steven Clarke.  What is an End User Software Engineer? , 2007, End-User Software Engineering.

[3]  Marian Petre,et al.  Cognitive dimensions 'beyond the notation' , 2006, J. Vis. Lang. Comput..

[4]  Richard Baskerville,et al.  Diversity in information systems action research methods , 1998 .

[5]  Karen Holtzblatt,et al.  Rapid Contextual Design: A How-To Guide to Key Techniques for User-Centered Design , 2004, UBIQ.

[6]  Karel Vredenburg,et al.  A survey of user-centered design practice , 2002, CHI.

[7]  Saleema Amershi,et al.  Designing for effective end-user interaction with machine learning , 2011, UIST '11 Adjunct.

[8]  Paul Mulholland,et al.  A cognitive dimensions analysis of interaction design for algorithmic composition software , 2014, PPIG.

[9]  Frank E. Ritter,et al.  Foundations for Designing User-Centered Systems: What System Designers Need to Know about People , 2014 .

[10]  Joseph J. LaViola,et al.  Exploring strategies and guidelines for developing full body video game interfaces , 2010, FDG.

[11]  Maya Cakmak,et al.  Power to the People: The Role of Humans in Interactive Machine Learning , 2014, AI Mag..

[12]  Mark Graham Brown,et al.  The Innovation Index , 2013 .

[13]  John M. Carroll,et al.  Artifact as theory-nexus: hermeneutics meets theory-based design , 1989, CHI '89.

[14]  HENRY LIEBERMAN,et al.  End-User Development: An Emerging Paradigm , 2006, End User Development.

[15]  Yoav Freund,et al.  A Short Introduction to Boosting , 1999 .

[16]  Marian Petre,et al.  Usability Analysis of Visual Programming Environments: A 'Cognitive Dimensions' Framework , 1996, J. Vis. Lang. Comput..

[17]  E. Hippel.  Sticky Information and the Locus of Problem Solving: Implications for Innovation , 1994 .

[18]  Steven Flowers,et al.  User Innovation in the music software industry: the case of Sibelius , 2015 .

[19]  P. Braunerhjelm,et al.  The knowledge spillover theory of intrapreneurship , 2018 .

[20]  Matthew Turk,et al.  A Random Walk through Eigenspace , 2001 .

[21]  Aditya G. Parameswaran,et al.  How Developers Iterate on Machine Learning Workflows - A Survey of the Applied Machine Learning Literature , 2018, ArXiv.

[22]  Eric von Hippel,et al.  User toolkits for innovation , 2001 .

[23]  Esteban Maestre,et al.  repoVizz: a framework for remote storage, browsing, annotation, and exchange of multi-modal data , 2013, MM '13.

[24]  Steven Clarke.  How Usable Are Your APIs? , 2011, Making Software.

[25]  Michael Zbyszynski,et al.  Write once run anywhere revisited: machine learning and audio tools in the browser with C++ and emscripten , 2017 .

[26]  Ralph Johnson,et al.  design patterns elements of reusable object oriented software , 2019 .

[27]  Beryl Plimmer,et al.  Designing an API at an appropriate abstraction level for programming social robot applications , 2017, J. Vis. Lang. Comput..

[28]  Nicu Sebe,et al.  Multimodal Human Computer Interaction: A Survey , 2005, ICCV-HCI.

[29]  Atau Tanaka,et al.  HapticWave: Presenting the Multiple Voices, Artefacts and Materials of a Design Research Project , 2015 .

[30]  R. Rapoport.  Three Dilemmas in Action Research , 1970 .

[31]  Norbert Schnell,et al.  Probabilistic Models for Designing Motion and Sound Relationships , 2014, NIME.

[32]  Alan Cooper,et al.  About Face 3: the essentials of interaction design , 1995 .

[33]  Dan R. Olsen,et al.  Evaluating user interface systems research , 2007, UIST.

[34]  Brad A. Myers,et al.  Past, Present and Future of User Interface Software Tools , 2000, TCHI.

[35]  Benjamin Recht,et al.  KeystoneML: Optimizing Pipelines for Large-Scale Advanced Analytics , 2016, 2017 IEEE 33rd International Conference on Data Engineering (ICDE).

[36]  Anna L. Cox,et al.  Questionnaires, in-depth interviews and focus groups , 2008 .

[37]  Francis K. H. Quek,et al.  Finding-NEVO: Toward Radical Design in HCI , 2013, INTERACT.

[38]  Ian H. Witten,et al.  Interactive machine learning: letting users build classifiers , 2002, Int. J. Hum. Comput. Stud..

[39]  Mark W. Newman,et al.  The challenges of user-centered design and evaluation for infrastructure , 2003, CHI '03.

[40]  Wanda J. Orlikowski,et al.  Research Commentary: Desperately Seeking the "IT" in IT Research - A Call to Theorizing the IT Artifact , 2001, Inf. Syst. Res..

[41]  Francisco Bernardo,et al.  O soli mio: exploring millimeter wave radar for musical interaction , 2017, NIME.

[42]  Lynn Conway,et al.  Introduction to VLSI systems , 1978 .

[43]  Eric von Hippel,et al.  Satisfying Heterogeneous User Needs Via Innovation Toolkits: The Case of Apache Security Software , 2002 .

[44]  Larry E. Wood,et al.  User Interface Design: Bridging the Gap from User Requirements to Design , 1997 .

[45]  Qian Yang,et al.  Grounding Interactive Machine Learning Tool Design in How Non-Experts Actually Build Models , 2018, Conference on Designing Interactive Systems.

[46]  Eric A. von Hippel,et al.  How Open Source Software Works: 'Free' User-to-User Assistance? , 2000 .

[47]  Maryann P. Feldman,et al.  R&D spillovers and recipient firm size , 1994 .

[48]  Eric von Hippel,et al.  Lead Users: An Important Source of Novel Product Concepts , 1986 .

[49]  Margaret M. Burnett,et al.  Using cognitive dimensions: Advice from the trenches , 2006, J. Vis. Lang. Comput..

[50]  Frédéric Bevilacqua,et al.  Supporting User Interaction with Machine Learning through Interactive Visualizations , 2016 .

[51]  Martín Abadi,et al.  TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems , 2016, ArXiv.

[52]  Scott R. Klemmer,et al.  Authoring sensor-based interactions by demonstration with direct manipulation and pattern recognition , 2007, CHI.

[53]  Jaroslav Tulach.  Practical API Design: Confessions of a Java Framework Architect , 2008 .

[54]  Yee ‐ King.  Collaborative Coding Interfaces on the Web , .

[55]  Robert M. Davison,et al.  Principles of canonical action research , 2004, Inf. Syst. J..

[56]  Pedro M. Domingos.  A few useful things to know about machine learning , 2012, Commun. ACM.

[57]  Holger M. Kienle.  Grounded innovation: strategies for creating digital products by Lars Erik Holmquist , 2013, SOEN.

[58]  David C. Mowery,et al.  Universities in National Innovation Systems , 2006 .

[59]  William W. Gaver.  What should we expect from research through design? , 2012, CHI.

[60]  Shumin Zhai,et al.  Technology Transfer of HCI Research Innovations: Challenges and Opportunities , 2015, CHI Extended Abstracts.

[61]  K. Nkrumah,et al.  Key research findings , 2010 .

[62]  Alan Cooper,et al.  The Inmates are Running the Asylum , 1999, Software-Ergonomie.

[63]  Nauman Bin Ali,et al.  Towards innovation measurement in the software industry , 2013, J. Syst. Softw..

[64]  William Buxton,et al.  Usability evaluation considered harmful (some of the time) , 2008, CHI.

[65]  eva Kühn,et al.  Automated measurement of API usability: The API Concepts Framework , 2015, Inf. Softw. Technol..

[66]  Colin Potts,et al.  Design of Everyday Things , 1988 .

[67]  Jill Slay,et al.  A Generic Cognitive Dimensions Questionnaire to Evaluate the Usability of Security APIs , 2017, HCI.

[68]  Brad Abrams,et al.  Framework Design Guidelines: Conventions, Idioms, and Patterns for Reusable .NET Libraries , 2005 .

[69]  Björn Hartmann,et al.  Machine Learning for Makers: Interactive Sensor Data Classification Based on Augmented Code Examples , 2017, Conference on Designing Interactive Systems.

[70]  Jan Gulliksen,et al.  User-centered System Design , 2011 .

[71]  Frank Maurer,et al.  What makes a good code example?: A study of programming Q&A in StackOverflow , 2012, 2012 28th IEEE International Conference on Software Maintenance (ICSM).

[72]  C. P. Goodman,et al.  The Tacit Dimension , 2003 .

[73]  Jerry Alan Fails,et al.  Interactive machine learning , 2003, IUI '03.

[74]  Andrew F. Monk.  Lightweight Techniques to Encourage Innovative User Interface Design , 2018 .

[75]  Atau Tanaka,et al.  Haptic Wave: A Cross-Modal Interface for Visually Impaired Audio Producers , 2016, CHI.

[76]  Susanne Bødker,et al.  Action Research: Its Nature and Relationship to Human-Computer Interaction by Ned Kock , 2011 .

[77]  R. Caruana,et al.  Structured Labeling to Facilitate Concept Evolution in Machine Learning , 2014 .

[78]  Lars Erik Holmquist,et al.  Bootlegging: multidisciplinary brainstorming with cut-ups , 2008, PDC.

[79]  James A. Landay,et al.  Investigating statistical machine learning as a tool for software development , 2008, CHI.

[80]  Perry R. Cook,et al.  Real-time human interaction with supervised learning algorithms for music composition and performance , 2011 .

[81]  Gillian R. Hayes.  The relationship of action research to human-computer interaction , 2011, TCHI.

[82]  Marina Apaydin,et al.  A Multi-Dimensional Framework of Organizational Innovation: A Systematic Review of the Literature , 2010 .

[83]  Leslie M. Blaha,et al.  Interface Metaphors for Interactive Machine Learning , 2017, HCI.

[84]  Jakob Nielsen,et al.  Usability engineering , 1997, The Computer Science and Engineering Handbook.

[85]  Aditya G. Parameswaran,et al.  Helix: Accelerating Human-in-the-loop Machine Learning , 2018, Proc. VLDB Endow..

[86]  Robert G. Cooper,et al.  Stage-gate systems: A new tool for managing new products , 1990 .

[87]  James A. Landay,et al.  Gestalt: integrated support for implementation and analysis in machine learning , 2010, UIST.

[88]  Atau Tanaka,et al.  User-centered design of a tool for interactive computer-generated audiovisuals , 2014 .

[89]  D. Ruppert.  The Elements of Statistical Learning: Data Mining, Inference, and Prediction , 2004 .

[90]  Celine Latulipe,et al.  Quantifying the Creativity Support of Digital Tools through the Creativity Support Index , 2014, ACM Trans. Comput. Hum. Interact..

[91]  Curtis Becker,et al.  Using the cognitive dimensions framework to measure the usability of a class library , 2003, PPIG.

[92]  Ana L. N. Fred,et al.  BITalino - A Multimodal Platform for Physiological Computing , 2013, ICINCO.

[93]  Alan F. Blackwell,et al.  A Cognitive Dimensions questionnaire optimised for users , 2000, PPIG.

[94]  Robert Watson.  Applying the Cognitive Dimensions of API Usability to Improve API Documentation Planning , 2014, SIGDOC.

[95]  Lars Frederiksen,et al.  Why Do Users Contribute to Firm-Hosted User Communities? The Case of Computer-Controlled Music Instruments , 2006, Organ. Sci..

[96]  Mark Bilandzic,et al.  Towards Participatory Action Design Research: Adapting Action Research and Design Science Research Methods for Urban Informatics , 2011, J. Community Informatics.

[97]  G. Susman,et al.  An Assessment of the Scientific Merits of Action Research. , 1978 .

[98]  Thomas R. G. Green,et al.  Cognitive dimensions of notations , 1990 .

[99]  Christopher Ré,et al.  Materialization optimizations for feature selection workloads , 2014, SIGMOD Conference.

[100]  Kim Halskov,et al.  Inspiration card workshops , 2006, DIS '06.

[101]  Per Ola Kristensson,et al.  A Review of User Interface Design for Interactive Machine Learning , 2018, ACM Trans. Interact. Intell. Syst..

[102]  Burr Settles,et al.  Active Learning Literature Survey , 2009 .

[103]  Jock D. Mackinlay,et al.  The design space of input devices , 1990, CHI '90.

[104]  Wendy E. Mackay,et al.  Using video to support interaction design , 2002 .

[105]  Perry R. Cook,et al.  Human model evaluation in interactive supervised learning , 2011, CHI.

[106]  Peter Norvig,et al.  Artificial Intelligence: A Modern Approach , 1995 .

[107]  David E. Avison,et al.  Controlling action research projects , 2001, Inf. Technol. People.

[108]  Donald A. Norman,et al.  The research-practice gap: the need for translational developers , 2010, INTR.

[109]  Eric Horvitz,et al.  Principles of mixed-initiative user interfaces , 1999, CHI '99.

[110]  Brad A. Myers,et al.  Improving API usability , 2016, Commun. ACM.

[111]  Jeff Axup,et al.  Participatory Design and Action Research: Identical Twins or Synergetic Pair? , 2006 .

[112]  Norbert Schnell,et al.  MnM: a Max/MSP mapping toolbox , 2005, NIME.

[113]  Norbert Schnell,et al.  A multimodal probabilistic model for gesture--based control of sound synthesis , 2013, MM '13.

[114]  Richard Baskerville,et al.  Investigating Information Systems with Action Research , 1999, Commun. Assoc. Inf. Syst..

[115]  David Maxwell Chickering,et al.  ModelTracker: Redesigning Performance Analysis Tools for Machine Learning , 2015, CHI.

[116]  E. Hippel,et al.  Measuring User Innovation in the UK: the importance of product creation by users , 2010 .

[117]  K. Lewin.  Action Research and Minority Problems , 1946 .

[118]  Chad Anthony Austin,et al.  Renaissance : A Functional Shading Language , 2005 .

[119]  Xavier Serra,et al.  Freesound technical demo , 2013, ACM Multimedia.

[120]  Francisco Bernardo,et al.  User-Centred Design Actions for Lightweight Evaluation of an Interactive Machine Learning Toolkit , 2018, Journal of Science and Technology of the Arts.

[121]  Alon Zakai.  Emscripten: an LLVM-to-JavaScript compiler , 2011, OOPSLA Companion.

[122]  Ned Kock,et al.  The Three Threats of Organizational Action Research , 2007 .

[123]  Michi Henning.  API: Design Matters , 2007, ACM Queue.

[124]  Nara L. Newcomer,et al.  User Centered Design , 2014, Encyclopedia of Database Systems.

[125]  Celine Latulipe,et al.  Dimensional Reasoning and Research Design Spaces , 2017, Creativity & Cognition.

[126]  Tim Brown,et al.  Change by Design: How Design Thinking Transforms Organizations and Inspires Innovation , 2009 .

[127]  C. Webb.  The Action Research Dissertation. A Guide for Students and Faculty , 2007 .

[128]  Atau Tanaka,et al.  Adaptive Gesture Recognition with Variation Estimation for Interactive Systems , 2014, ACM Trans. Interact. Intell. Syst..

[129]  Axel Röbel,et al.  MuBu and Friends - Assembling Tools for Content Based Real-Time Interactive Audio Processing in Max/MSP , 2009, ICMC.

[130]  John Millar Carroll.  HCI Models, Theories, and Frameworks: Toward a Multidisciplinary Science , 2003 .

[131]  Norbert Schnell,et al.  Continuous Realtime Gesture Following and Recognition , 2009, Gesture Workshop.

[132]  Desney S. Tan,et al.  EnsembleMatrix: interactive visualization to support machine learning with multiple classifiers , 2009, CHI.

[133]  Martin P. Robillard,et al.  A field study of API learning obstacles , 2011, Empirical Software Engineering.

[134]  Lars Kotthoff,et al.  Auto-WEKA 2.0: Automatic model selection and hyperparameter optimization in WEKA , 2017, J. Mach. Learn. Res..

[135]  Herbert A. Simon,et al.  The Structure of Ill Structured Problems , 1973, Artif. Intell..

[136]  Daniel A. Levinthal,et al.  ABSORPTIVE CAPACITY: A NEW PERSPECTIVE ON LEARNING AND INNOVATION , 1990 .

[137]  Qiang Yang,et al.  A Survey on Transfer Learning , 2010, IEEE Transactions on Knowledge and Data Engineering.

[138]  Silvia Lindtner,et al.  Emerging sites of HCI innovation: hackerspaces, hardware startups & incubators , 2014, CHI.

[139]  Joseph A. Paradiso,et al.  The gesture recognition toolkit , 2014, J. Mach. Learn. Res..

[140]  Ali Momeni,et al.  Ml.lib: robust, cross-platform, open-source machine learning for max and pure data , 2015, NIME.

[141]  Alan F. Blackwell,et al.  First steps in programming: a rationale for attention investment models , 2002, Proceedings IEEE 2002 Symposia on Human Centric Computing Languages and Environments.

[142]  Austin Henderson,et al.  Interaction design: beyond human-computer interaction , 2002, UBIQ.

[143]  John Cullen,et al.  Democratizing Innovation , 2020, Encyclopedia of Creativity, Invention, Innovation and Entrepreneurship.

[144]  Jacob O. Wobbrock,et al.  From User-Centered to Adoption-Centered Design: A Case Study of an HCI Research Innovation Becoming a Product , 2015, CHI.

[145]  Ben Shneiderman,et al.  Design Principles for Tools to Support Creative Thinking , 2005 .

[146]  Anind K. Dey,et al.  a CAPpella: programming by demonstration of context-aware applications , 2004, CHI.