Wearable Finger Tracking and Cutaneous Haptic Interface with Soft Sensors for Multi-Fingered Virtual Manipulation

Multi-Fingered haptics is imperative for truly immersive virtual reality experience, as many real-world tasks involve finger manipulation. One of the key lacking aspect for this is the absence of technologically and economically viable wearable haptic interfaces that can simultaneously track the finger/hand motions and display multi-degree-of-freedom (DOF) contact forces. In this paper, we propose a novel wearable cutaneous haptic interface (WCHI), which consists of 1) finger tracking modules (FTMs) to estimate complex multi-DOF finger and hand motion; and 2) cutaneous haptic modules (CHMs) to convey three-DOF contact force at the finger-tip. By opportunistically utilizing such different types of sensors as inertial measurement units, force sensitive resistor sensors, and soft sensors, the WCHI can track complex anatomically consistent multi-DOF finger motion while avoiding FTM-CHM electromagnetic interference possibly stemming from their collocation in the small form-factor interface; while also providing the direction and magnitude of three-DOF finger-tip contact force, the feedback of which can significantly enhance the precision of contact force generation against variability among users via their closed-loop control. Human subject study is performed with a virtual peg insertion task to show the importance of both the multi-DOF finger tracking and the three-DOF cutaneous haptic feedback for dexterous manipulation in virtual environment.

[1]  Y. Matsuoka,et al.  Mechanisms of the Anatomically Correct Testbed Hand , 2013, IEEE/ASME Transactions on Mechatronics.

[2]  Jan Peters,et al.  A low-cost sensor glove with vibrotactile feedback and multiple finger joint and hand motion sensing for human-robot interaction , 2016, 2016 25th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN).

[3]  Allison M. Okamura,et al.  Three-Dimensional Skin Deformation as Force Substitution: Wearable Device Design and Performance During Haptic Exploration of Virtual Environments , 2017, IEEE Transactions on Haptics.

[4]  Claudio Pacchierotti,et al.  Wearable haptics and hand tracking via an RGB-D camera for immersive tactile experiences , 2014, SIGGRAPH '14.

[5]  F. Markley.  Attitude determination using vector observations and the singular value decomposition , 1988 .

[6]  Nancy S. Pollard,et al.  Method for Determining Kinematic Parameters of the In Vivo Thumb Carpometacarpal Joint , 2008, IEEE Transactions on Biomedical Engineering.

[7]  Antonis A. Argyros,et al.  Efficient model-based 3D tracking of hand articulations using Kinect , 2011, BMVC.

[8]  Suin Kim,et al.  A Soft Sensor-Based Three-Dimensional (3-D) Finger Motion Measurement System , 2017, Sensors.

[9]  John L. Salmon,et al.  Capabilities of Current Generation Virtual Reality to Enhance the Design Process , 2017 .

[10]  Kevin J. MacKenzie,et al.  Finger length and distal finger extent patterns in humans. , 2002, American journal of physical anthropology.

[11]  Dongjun Lee,et al.  Enlarging just noticeable differences of visual-proprioceptive conflict in VR using haptic feedback , 2015, 2015 IEEE World Haptics Conference (WHC).

[12]  Wan Kyun Chung,et al.  Human kinematic factor for haptic manipulation : the wrist to thumb , 2002, Proceedings 10th Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems. HAPTICS 2002.

[13]  Sang Wook Lee,et al.  Improving Kinematic Accuracy of Soft Wearable Data Gloves by Optimizing Sensor Locations , 2016, Sensors.

[14]  Jian S. Dai,et al.  An Optimization Approach to Teleoperation of the Thumb of a Humanoid Robot Hand: Kinematic Mapping and Calibration , 2014 .

[15]  Claudio Pacchierotti,et al.  Evaluation of Wearable Haptic Systems for the Fingers in Augmented Reality Applications , 2017, IEEE Transactions on Haptics.

[16]  Allison M. Okamura,et al.  Sensory Substitution and Augmentation Using 3-Degree-of-Freedom Skin Deformation Feedback , 2015, IEEE Transactions on Haptics.

[17]  Richard M. Murray,et al.  A Mathematical Introduction to Robotic Manipulation , 1994 .

[18]  Claudio Pacchierotti,et al.  Cutaneous Force Feedback as a Sensory Subtraction Technique in Haptics , 2011, IEEE Transactions on Haptics.

[19]  Kouta Minamizawa,et al.  Gravity grabber: wearable haptic display to present virtual mass sensation , 2007, SIGGRAPH '07.

[20]  Robert E. Mahony,et al.  Nonlinear Complementary Filters on the Special Orthogonal Group , 2008, IEEE Transactions on Automatic Control.

[21]  Yoky Matsuoka,et al.  A kinematic thumb model for the ACT hand , 2006, Proceedings 2006 IEEE International Conference on Robotics and Automation, 2006. ICRA 2006..

[22]  L. M. Myers,et al.  The axes of rotation of the thumb interphalangeal and metacarpophalangeal joints. , 1995, Clinical orthopaedics and related research.

[23]  Yiwei Tao,et al.  Wearable soft artificial skin for hand motion detection with embedded microfluidic strain sensing , 2015, 2015 IEEE International Conference on Robotics and Automation (ICRA).

[24]  Domenico Prattichizzo,et al.  Using inertial and magnetic sensors for hand tracking and rendering in wearable haptics , 2015, 2015 IEEE World Haptics Conference (WHC).

[25]  Jun-Sik Kim,et al.  Physics-based hand interaction with virtual objects , 2015, 2015 IEEE International Conference on Robotics and Automation (ICRA).

[26]  Claudio Pacchierotti,et al.  Towards Wearability in Fingertip Haptics: A 3-DoF Wearable Device for Cutaneous Force Feedback , 2013, IEEE Transactions on Haptics.

[27]  Claudio Pacchierotti,et al.  Using a Fingertip Tactile Device to Substitute Kinesthetic Feedback in Haptic Interaction , 2010, EuroHaptics.

[28]  Dongjun Lee,et al.  Haptic rendering and interactive simulation using passive midpoint integration , 2017, Int. J. Robotics Res..

[29]  Dongjun Lee,et al.  On utilizing pseudo-haptics for cutaneous fingertip haptic device , 2014, 2014 IEEE Haptics Symposium (HAPTICS).

[30]  Antonio Bicchi,et al.  Low-cost, fast and accurate reconstruction of robotic and human postures via IMU measurements , 2015, 2015 IEEE International Conference on Robotics and Automation (ICRA).

[31]  Antonio Frisoli,et al.  A wearable fingertip haptic device with 3 DoF asymmetric 3-RSR kinematics , 2015, 2015 IEEE World Haptics Conference (WHC).

[32]  Francesco Chinello,et al.  Efficient nonlinear skin simulation for multi-finger tactile rendering , 2016, 2016 IEEE Haptics Symposium (HAPTICS).

[33]  Mathias Wilhelm,et al.  Whole hand modeling using 8 wearable sensors: biomechanics for hand pose prediction , 2013, AH.

[34]  Gerd Hirzinger,et al.  The thumb: guidelines for a robotic design , 2010, 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[35]  Yong-Lae Park,et al.  Design and Fabrication of Soft Artificial Skin Using Embedded Microchannels and Liquid Conductors , 2012, IEEE Sensors Journal.

[36]  Domenico Prattichizzo,et al.  Fingertip force estimation via inertial and magnetic sensors in deformable object manipulation , 2016, 2016 IEEE Haptics Symposium (HAPTICS).

[37]  John L. Salmon,et al.  A Review of the Capabilities of Current Low-Cost Virtual Reality Technology and Its Potential to Enhance the Design Process , 2017, J. Comput. Inf. Sci. Eng..