“I want that”: Human-in-the-loop control of a wheelchair-mounted robotic arm

Wheelchair-mounted robotic arms have been commercially available for a decade. In order to operate these robotic arms, a user must have a high level of cognitive function. Our research focuses on replacing a manufacturer-provided, menu-based interface with a vision-based system while adding autonomy to reduce the cognitive load. Instead of manual task decomposition and execution, the user explicitly designates the end goal, and the system autonomously retrieves the object. In this paper, we present the complete system which can autonomously retrieve a desired object from a shelf. We also present the results of a 15-week study in which 12 participants from our target population used our system, totaling 198 trials.

[1]  H.J.A. Stuyt,et al.  Cost-savings and economic benefits due to the assistive robotic manipulator (ARM) , 2005, 9th International Conference on Rehabilitation Robotics, 2005. ICORR 2005..

[2]  C. Leroux,et al.  Intuitive human interaction with an arm robot for severely handicapped people - A One Click Approach , 2007, 2007 IEEE 10th International Conference on Rehabilitation Robotics.

[3]  M. Lorr,et al.  Two EITS Manual for the Profile of Mood States (1971 & 1992) , 1971 .

[4]  Jean-Yves Bouguet,et al.  Camera calibration toolbox for matlab , 2001 .

[5]  Robert C. Bolles,et al.  Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography , 1981, CACM.

[6]  Warren E. Dixon,et al.  Adaptive 2.5D visual servoing of kinematically redundant robot manipulators , 2002, Proceedings of the 41st IEEE Conference on Decision and Control, 2002..

[7]  H. Nguyen.  El-E: An Assistive Robot that Fetches Objects from Flat Surfaces , 2008 .

[8]  Dae-Jin Kim,et al.  Motion Segmentation and Control Design for UCF-MANUS—An Intelligent Assistive Robotic Manipulator , 2012, IEEE/ASME Transactions on Mechatronics.

[9]  Vincent Lepetit,et al.  Fast Keypoint Recognition in Ten Lines of Code , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[10]  Keinosuke Fukunaga,et al.  Introduction to Statistical Pattern Recognition , 1972 .

[11]  M. Lorr,et al.  Profile of mood states , 1971 .

[12]  Robert Sessions Woodworth,et al.  THE ACCURACY OF VOLUNTARY MOVEMENT , 1899 .

[13]  J. Eldredge.  Rehabilitation of the Head Injured Adult , 1984 .

[14]  G. G. Stokes.  "J." , 1890, The New Yale Book of Quotations.

[15]  Matthijs C. Dorst.  Distinctive Image Features from Scale-Invariant Keypoints , 2011 .

[16]  Warren E. Dixon,et al.  Adaptive homography-based visual servo tracking for a fixed camera configuration with a camera-in-hand extension , 2005, IEEE Transactions on Control Systems Technology.

[17]  Constantine Stephanidis,et al.  Universal access in the information society , 1999, HCI.

[18]  Claire Dune,et al.  Active rough shape estimation of unknown objects , 2008, 2008 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[19]  Charles C. Kemp,et al.  Laser pointers and a touch screen: intuitive interfaces for autonomous mobile manipulation for the motor impaired , 2008, Assets '08.

[20]  H. Nguyen,et al.  Bio-inspired assistive robotics: Service dogs as a model for human-robot interaction and mobile manipulation , 2008, 2008 2nd IEEE RAS & EMBS International Conference on Biomedical Robotics and Biomechatronics.

[21]  David Nistér,et al.  Scalable Recognition with a Vocabulary Tree , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[22]  Dae-Jin Kim,et al.  Eye-in-hand stereo visual servoing of an assistive robot arm in unstructured environments , 2009, 2009 IEEE International Conference on Robotics and Automation.

[23]  William Harwin,et al.  Devices for assisting manipulation: a summary of user task priorities , 1994 .

[24]  Claire Dune,et al.  Vision-based grasping of unknown objects to improve disabled persons autonomy , 2008 .

[25]  Katherine M. Tsui,et al.  Developing heuristics for assistive robotics , 2010, HRI 2010.

[26]  O. Faugeras.  Three-dimensional computer vision: a geometric viewpoint , 1993 .

[27]  Keinosuke Fukunaga,et al.  Introduction to statistical pattern recognition (2nd ed.) , 1990 .

[28]  J. A. van Woerden,et al.  Collaborative Control of the Manus Manipulator , 2005, 9th International Conference on Rehabilitation Robotics, 2005. ICORR 2005..

[29]  J. Jutai,et al.  Psychosocial Impact of Assistive Devices Scale (PIADS) , 2002 .

[30]  Katherine M. Tsui,et al.  Development and evaluation of a flexible interface for a wheelchair mounted robotic arm , 2008, 2008 3rd ACM/IEEE International Conference on Human-Robot Interaction (HRI).