Engineering human-in-the-loop interactions in cyber-physical systems

Abstract Context: Cyber-Physical Systems (CPSs) are gradually and widely introducing autonomous capabilities into everything. However, human participation is required to accomplish tasks that are better performed with humans (often called human-in-the-loop). In this way, human-in-the-loop solutions have the potential to handle complex tasks in unstructured environments, by combining the cognitive skills of humans with autonomous systems behaviors. Objective: The objective of this paper is to provide appropriate techniques and methods to help designers analyze and design human-in-the-loop solutions. These solutions require interactions that engage the human, provide natural and understandable collaboration, and avoid disturbing the human in order to improve human experience. Method: We have analyzed several works that identified different requirements and critical factors that are relevant to the design of human-in-the-loop solutions. Based on these works, we have defined a set of design principles that are used to build our proposal. Fast-prototyping techniques have been applied to simulate the designed human-in-the-loop solutions and validate them. Results: We have identified the technological challenges of designing human-in-the-loop CPSs and have provided a method that helps designers to identify and specify how the human and the system should work together, focusing on the control strategies and interactions required. Conclusions: The use of our approach facilitates the design of human-in-the-loop solutions. Our method is practical at earlier stages of the software life cycle since it allows domain experts to focus on the problem and not on the solution.

[1]  Vicente Pelechano,et al.  Personalization for unobtrusive service interaction , 2012, Personal and Ubiquitous Computing.

[2]  Erik Hollnagel,et al.  Principles for modelling function allocation , 2000, Int. J. Hum. Comput. Stud..

[3]  Manfred Tscheligi,et al.  Control Transition Interfaces in Semiautonomous Vehicles: A Categorization Framework and Literature Analysis , 2017, AutomotiveUI.

[4]  Jakob Nielsen,et al.  A mathematical model of the finding of usability problems , 1993, INTERCHI.

[5]  Yu-Chee Tseng,et al.  iMouse: An Integrated Mobile Surveillance and Wireless Sensor System , 2007, Computer.

[6]  Jorge Sá Silva,et al.  A Survey on Human-in-the-Loop Applications Towards an Internet of All , 2015, IEEE Communications Surveys & Tutorials.

[7]  Mladjan Jovanovic,et al.  Bridging User Context and Design Models to Build Adaptive User Interfaces , 2014, HCSE.

[8]  Sirajum Munir,et al.  Cyber Physical System Challenges for Human-in-the-Loop Control , 2013, Feedback Computing.

[9]  Julio Abascal,et al.  Universal accessibility as a multimodal design issue , 2007, CACM.

[10]  Claes Wohlin,et al.  Experimentation in software engineering: an introduction , 2000 .

[11]  David E. Kieras,et al.  The GOMS family of user interface analysis techniques: comparison and contrast , 1996, TCHI.

[12]  Ellen J. Bass,et al.  Using Formal Verification to Evaluate Human-Automation Interaction: A Review , 2013, IEEE Transactions on Systems, Man, and Cybernetics: Systems.

[13]  Bob Fields,et al.  Function allocation: a perspective from studies of work practice , 2000, Int. J. Hum. Comput. Stud..

[14]  Anind K. Dey,et al.  Understanding and Using Context , 2001, Personal and Ubiquitous Computing.

[15]  Mohan M. Trivedi,et al.  Holistic Sensing and Active Displays for Intelligent Driver Support Systems , 2007, Computer.

[16]  William H. Sanders,et al.  The Multiple-Asymmetric-Utility System Model: A Framework for Modeling Cyber-Human Systems , 2011, 2011 Eighth International Conference on Quantitative Evaluation of SysTems.

[17]  Katrina D. Maxwell,et al.  Applied Statistics for Software Managers , 2002 .

[18]  John D. Lee,et al.  Trust in Automation: Designing for Appropriate Reliance , 2004 .

[19]  Paul P. Maglio,et al.  Dealing with ghosts: Managing the user experience of autonomic computing , 2003, IBM Syst. J..

[20]  Natalia Juristo Juzgado,et al.  Basics of Software Engineering Experimentation , 2010, Springer US.

[21]  S. Shankar Sastry,et al.  Synthesis for Human-in-the-Loop Control Systems , 2014, TACAS.

[22]  Bako Rajaonah,et al.  The role of intervening variables in driver-ACC cooperation , 2008, Int. J. Hum. Comput. Stud..

[23]  Arne Jönsson,et al.  Wizard of Oz studies -- why and how , 1993, Knowl. Based Syst..

[24]  Lewis L. Chuang,et al.  Looming Auditory Collision Warnings for Semi-Automated Driving: An ERP Study , 2018, AutomotiveUI.

[25]  Ufuk Topcu,et al.  Synthesis of Human-in-the-Loop Control Protocols for Autonomous Systems , 2016, IEEE Transactions on Automation Science and Engineering.

[26]  Shihong Huang,et al.  Incorporating Human Intention into Self-Adaptive Systems , 2015, 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering.

[27]  Wendy Ju,et al.  The Design of Implicit Interactions: Making Interactive Systems Less Obnoxious , 2008, Design Issues.

[28]  In Lee,et al.  Efficacy of haptic blind spot warnings applied through a steering wheel or a seatbelt , 2013 .

[29]  Andrew L. Kun,et al.  History and future of human-automation interaction , 2019, Int. J. Hum. Comput. Stud..

[30]  Thomas B. Sheridan,et al.  Function allocation: algorithm, alchemy or apostasy? , 2000, Int. J. Hum. Comput. Stud..

[31]  Joëlle Coutaz,et al.  Plasticity of User Interfaces: Framework and Research Agenda , 1999, INTERACT.

[32]  Peter Johnson,et al.  Tasks and Situations: Considerations for Models and Design Principles in Human Computer Interaction , 1999, HCI.

[33]  James A. Larson,et al.  Guidelines for multimodal user interface design , 2004, CACM.

[34]  S. Jamieson Likert scales: how to (ab)use them , 2004, Medical education.

[35]  Gilbert L. Peterson,et al.  A function-to-task process model for adaptive automation system design , 2014, Int. J. Hum. Comput. Stud..

[36]  Shihong Huang,et al.  Improving Human-in-the-Loop Adaptive Systems Using Brain-Computer Interaction , 2017, 2017 IEEE/ACM 12th International Symposium on Software Engineering for Adaptive and Self-Managing Systems (SEAMS).

[37]  Jean Vanderdonckt,et al.  A Unifying Reference Framework for multi-target user interfaces , 2003, Interact. Comput..

[38]  Andrew Shepherd,et al.  Hierarchical task analysis , 2001 .

[39]  Todd Litman,et al.  Autonomous Vehicle Implementation Predictions: Implications for Transport Planning , 2015 .

[40]  Jakob E. Bardram Distributed interaction , 2012, EICS '12.

[41]  Luís Carriço,et al.  Low-fi prototyping for mobile devices , 2006, CHI Extended Abstracts.

[42]  Anthony Jameson,et al.  User Modeling Meets Usability Goals , 2005, User Modeling.

[43]  T J Triggs,et al.  REACTION TIME OF DRIVERS TO ROAD STIMULI , 1982 .

[44]  Anton Nijholt,et al.  Modality effects on cognitive load and performance in high-load information presentation , 2009, IUI.

[45]  Richard N. Taylor,et al.  Coupling software architecture and human architecture for collaboration-aware system adaptation , 2013, 2013 35th International Conference on Software Engineering (ICSE).

[46]  Lorrie Faith Cranor,et al.  A Framework for Reasoning About the Human in the Loop , 2008, UPSEC.

[47]  Dari Trendafilov,et al.  Designing and evaluating multimodal interaction for mobile contexts , 2008, ICMI '08.

[48]  Allen Newell,et al.  The psychology of human-computer interaction , 1983 .

[49]  Apostolos P. Fournaris,et al.  Enabling the human in the loop: Linked data and knowledge in industrial cyber-physical systems , 2019, Annu. Rev. Control..

[50]  Laine Mears,et al.  A complementary Cyber-Human Systems framework for Industry 4.0 Cyber-Physical Systems , 2018 .

[51]  Eric Horvitz,et al.  Models of attention in computing and communication , 2003, Commun. ACM.

[52]  Regina A. Pomranky,et al.  The role of trust in automation reliance , 2003, Int. J. Hum. Comput. Stud..

[53]  Vicente Pelechano,et al.  Designing human-in-the-loop autonomous Cyber-Physical Systems , 2019, Int. J. Hum. Comput. Stud..

[54]  Per Runeson,et al.  Using Students as Experiment Subjects – An Analysis on Graduate and Freshmen Student Data , 2003 .

[55]  Andrew W. Moore,et al.  Four Thought Leaders on Where the Industry Is Headed , 2016, IEEE Softw..

[56]  Daniel L. Moody,et al.  The method evaluation model: a theoretical model for validating information systems design methods , 2003, ECIS.

[57]  B. Kitchenham,et al.  Case Studies for Method and Tool Evaluation , 1995, IEEE Softw..

[58]  Shiyan Hu,et al.  Design Automation of Cyber-Physical Systems: Challenges, Advances, and Opportunities , 2017, IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems.

[59]  Jonathan Grudin,et al.  Human-computer integration , 2016, Interactions.

[60]  Raja Parasuraman,et al.  Designing for Flexible Interaction Between Humans and Automation: Delegation Interfaces for Supervisory Control , 2007, Hum. Factors.

[61]  Deniz Erdogmus,et al.  The Future of Human-in-the-Loop Cyber-Physical Systems , 2013, Computer.

[62]  Pattie Maes,et al.  Robotic Symbionts: Interweaving Human and Machine Actions , 2018, IEEE Pervasive Computing.

[63]  Saeid Nahavandi,et al.  Trusted Autonomy Between Humans and Robots: Toward Human-on-the-Loop in Robotics and Autonomous Systems , 2017, IEEE Systems, Man, and Cybernetics Magazine.

[64]  Petr Fiedler,et al.  Human-in-the-Loop Issue in Context of the Cyber-Physical Systems , 2018 .

[65]  Niels Ole Bernsen,et al.  Foundations of Multimodal Representations: A Taxonomy of Representational Modalities , 1994, Interact. Comput..

[66]  Mark Vollrath,et al.  Distractive or Supportive -- How Warnings in the Head-up Display Affect Drivers' Gaze and Driving Behavior , 2015, 2015 IEEE 18th International Conference on Intelligent Transportation Systems.

[67]  Mike Kuniavsky,et al.  Smart Things: Ubiquitous Computing User Experience Design , 2010 .

[68]  Remo M A van der Heiden,et al.  Susceptibility to audio signals during autonomous driving , 2018, PloS one.

[69]  Anind K. Dey,et al.  Evaluating Intelligibility Usage and Usefulness in a Context-Aware Application , 2013, HCI.

[70]  Weng-Keen Wong,et al.  End-user interactions with intelligent and autonomous systems , 2012, CHI Extended Abstracts.

[71]  Eric Simmon,et al.  Cyber-Physical-Human Systems: Putting People in the Loop , 2016, IT Professional.