A POMDP framework for human-in-the-loop system

Human operators are involved in many real world systems such as automobile systems. Traditional human-assistance features such as warning systems in the aircraft and automatic braking systems in automobile only monitor the states of the machine in order to prevent human errors and enhance safety. We believe that next generation systems should be able to monitor both the human and the machine and give an appropriate feedback to them. Although having human in the control loop has its advantage, it lacks a unified modeling framework to manage the feedback between the human and the machine. In this paper, we will present how partially observable Markov decision process (POMDP) can be used as a unified framework for the three main components in a human-in-the-loop control system-the human model, the machine dynamic model and the observation model. We use simulations to show the benefits of this framework. Finally, we outline the key challenge to advance this framework.

[1]  Alberto L. Sangiovanni-Vincentelli,et al.  Data-Driven Probabilistic Modeling and Verification of Human Driver Behavior , 2014, AAAI Spring Symposia.

[2]  Nikos A. Vlassis,et al.  Perseus: Randomized Point-based Value Iteration for POMDPs , 2005, J. Artif. Intell. Res..

[3]  Christopher D. Wickens,et al.  A model for types and levels of human interaction with automation , 2000, IEEE Trans. Syst. Man Cybern. Part A.

[4]  Sirajum Munir,et al.  Cyber Physical System Challenges for Human-in-the-Loop Control , 2013, Feedback Computing.

[5]  Steve J. Young,et al.  Partially observable Markov decision processes for spoken dialog systems , 2007, Comput. Speech Lang..

[6]  Magnus Egerstedt,et al.  Human-in-the-loop: MPC for shared control of a quadruped rescue robot , 2011, 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[7]  Martin Buss,et al.  An HMM approach to realistic haptic human-robot interaction , 2009, World Haptics 2009 - Third Joint EuroHaptics conference and Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems.

[8]  Ruzena Bajcsy,et al.  Safe semi-autonomous control with enhanced driver modeling , 2012, 2012 American Control Conference (ACC).

[9]  Sterling J. Anderson,et al.  An optimal-control-based framework for trajectory planning, threat assessment, and semi-autonomous control of passenger vehicles in hazard avoidance scenarios , 2010 .

[10]  Alex Pentland,et al.  Modeling and Prediction of Human Behavior , 1999, Neural Computation.

[11]  Reid G. Simmons,et al.  Point-Based POMDP Algorithms: Improved Analysis and Implementation , 2005, UAI.

[12]  Weihua Sheng,et al.  Wearable Sensor-Based Hand Gesture and Daily Activity Recognition for Robot-Assisted Living , 2011, IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans.

[13]  Illah R. Nourbakhsh,et al.  Planning for Human–Robot Interaction in Socially Situated Tasks , 2013, Int. J. Soc. Robotics.

[14]  P. Poupart.  Exploiting structure to efficiently solve large scale partially observable Markov decision processes , 2005 .

[15]  Joelle Pineau,et al.  Point-based value iteration: An anytime algorithm for POMDPs , 2003, IJCAI.

[16]  Jesse Hoey,et al.  Automated handwashing assistance for persons with dementia using video and a partially observable Markov decision process , 2010, Comput. Vis. Image Underst..

[17]  Allison M. Okamura,et al.  Recognition of operator motions for real-time assistance using virtual fixtures , 2003, 11th Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems, 2003. HAPTICS 2003. Proceedings..

[18]  Brian C Tefft.  Asleep at the Wheel: The Prevalence and Impact of Drowsy Driving , 2010 .

[19]  Yoshihiko Nakamura,et al.  Recognition of human driving behaviors based on stochastic symbolization of time series signal , 2008, 2008 IEEE/RSJ International Conference on Intelligent Robots and Systems.