Kinect-based human finger tracking method for natural haptic rendering

Abstract In the multi-modal natural human-computer interaction (HCI), real-time finger position detection with high accuracy becomes an important basis for interactive modeling of the virtual environment. In this paper, a novel fingertip auto-positioning (FAP) detection method is proposed for tracking finger position with a Microsoft Kinect sensor. Based on the finger skeleton point obtained through the Kinect SDK (Software Development Kit), skeleton point correction and fingertip auto-positioning are realized with the pixels circle which covers the fingertip area within a certain threshold. The experimental results show that the average AMPE (Absolute Mean Percentage Error) of the proposed finger position detection method are 1.47%, 1.62% and 0.80% respectively in the direction of X, Y and Z axes while its execution rate is 23 Hz. The proposed method can be applied to haptic based virtual reality applications and provide required position information for haptic modeling.

[1]  Susan M. Astley,et al.  Evaluation of Kinect 3D Sensor for Healthcare Imaging , 2016, Journal of medical and biological engineering.

[2]  Debi Prosad Dogra,et al.  A multimodal framework for sensor based sign language recognition , 2017, Neurocomputing.

[3]  Nicolas Vuillerme,et al.  Real-Time Obstacle Detection System in Indoor Environment for the Visually Impaired Using Microsoft Kinect Sensor , 2016, J. Sensors.

[4]  Chanjira Sinthanayothin,et al.  Skeleton Tracking using Kinect Sensor & Displaying in 3D Virtual Scene , 2012 .

[5]  Pengwen Xiong,et al.  Visual-Haptic Aid Teleoperation Based on 3-D Environment Modeling and Updating , 2016, IEEE Transactions on Industrial Electronics.

[6]  Jeffrey M. Hausdorff,et al.  Validation of a Method for Real Time Foot Position and Orientation Tracking With Microsoft Kinect Technology for Use in Virtual Reality and Treadmill Based Gait Training Programs , 2014, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[7]  John Kenneth Salisbury,et al.  Haptic Rendering: Introductory Concepts , 2004, IEEE Computer Graphics and Applications.

[8]  Hervé Delingette,et al.  Improving realism of a surgery simulator: linear anisotropic elasticity, complex interactions and force extrapolation , 2002, Comput. Animat. Virtual Worlds.

[9]  Yoshifumi Kitamura,et al.  IM6D: magnetic tracking system with 6-DOF passive markers for dexterous 3D interaction and motion , 2015, ACM Trans. Graph..

[10]  Diego Alonso,et al.  A Machine Learning Approach to Pedestrian Detection for Autonomous Vehicles Using High-Definition 3D Range Data , 2016, Sensors.

[11]  J. Goetz.  Sensors That Can Take the Heat, Part 3: Design Techniques for High-Temperature Applications , 2000 .

[12]  Tilak Dutta,et al.  Evaluation of the Kinect™ sensor for 3-D kinematic measurement in the workplace. , 2012, Applied ergonomics.

[13]  Lin Yang,et al.  3-D Markerless Tracking of Human Gait by Geometric Trilateration of Multiple Kinects , 2018, IEEE Systems Journal.

[14]  Seongah Chin,et al.  Structural Motion Grammar for Universal Use of Leap Motion: Amusement and Functional Contents Focused , 2018, J. Sensors.

[15]  Adam Wojciechowski,et al.  Hybrid Orientation Based Human Limbs Motion Tracking Method , 2017, Sensors.

[16]  Cagatay Basdogan,et al.  Haptics in virtual environments: taxonomy, research status, and challenges , 1997, Comput. Graph..

[17]  Akira Mita,et al.  Markerless Knee Joint Position Measurement Using Depth Data during Stair Walking , 2017, Sensors.

[18]  Jiu-Jenq Lin,et al.  Measurement of scapular medial border and inferior angle prominence using a novel scapulometer: A reliability and validity study. , 2017, Musculoskeletal science & practice.

[19]  Aiguo Song,et al.  A Novel Human-Robot Cooperative Method for Upper Extremity Rehabilitation , 2017, Int. J. Soc. Robotics.

[20]  Harry Shum,et al.  Superpixel-based color–depth restoration and dynamic environment modeling for Kinect-assisted image-based rendering systems , 2016, The Visual Computer.

[21]  Gilson A. Giraldi,et al.  Hand gesture recognition from depth and infrared Kinect data for CAVE applications interaction , 2017, Multimedia Tools and Applications.

[22]  Ing-Jr Ding,et al.  Kinect microphone array-based speech and speaker recognition for the exhibition control of humanoid robots , 2017, Comput. Electr. Eng..

[23]  John E. Angus,et al.  Application of extended Kalman filter for improving the accuracy and smoothness of Kinect skeleton-joint estimates , 2014 .

[24]  Luiz Velho,et al.  Kinect and RGBD Images: Challenges and Applications , 2012, 2012 25th SIBGRAPI Conference on Graphics, Patterns and Images Tutorials.

[25]  Chen Feng,et al.  Upper limb motion tracking with the integration of IMU and Kinect , 2015, Neurocomputing.