Evaluating RGB+D hand posture detection methods for mobile 3D interaction

In mobile applications it is crucial to provide intuitive means for 2D and 3D interaction. A large number of techniques exist to support a natural user interface (NUI) by detecting the user's hand posture in RGB+D (depth) data. Depending on a given interaction scenario, each technique has its advantages and disadvantages. To evaluate the performance of the various techniques on a mobile device, we conducted a systematic study by comparing the accuracy of five common posture recognition approaches with varying illumination and background. To be able to perform this study, we developed a powerful software framework that is capable of processing and fusing RGB and depth data directly on a handheld device. Overall results reveal best recognition rate of posture detection for combined RGB+D data at the expense of update rate. Finally, to support users in choosing the appropriate technique for their specific mobile interaction task, we derived guidelines based on our study.

[1]  Tobias Höllerer,et al.  Handy AR: Markerless Inspection of Augmented Reality Objects Using Fingertip Tracking , 2007, 2007 11th IEEE International Symposium on Wearable Computers.

[2]  Hyotaek Lim,et al.  Hand tracking and gesture recognition system for human-computer interaction using low-cost hardware , 2015, Multimedia Tools and Applications.

[3]  Luc Van Gool,et al.  Combining RGB and ToF cameras for real-time 3D hand gesture interaction , 2011, WACV.

[4]  Rainer Lienhart,et al.  An extended set of Haar-like features for rapid object detection , 2002, Proceedings. International Conference on Image Processing.

[5]  Rini Akmeliawati,et al.  A hybrid method using haar-like and skin-color algorithm for hand posture detection, recognition and tracking , 2010, 2010 IEEE International Conference on Mechatronics and Automation.

[6]  Peter Fröhlich,et al.  Markerless visual fingertip detection for natural mobile device interaction , 2011, Mobile HCI.

[7]  Artzai Picón,et al.  Robust vision-based hand tracking using single camera for ubiquitous 3D gesture interaction , 2010, 2010 IEEE Symposium on 3D User Interfaces (3DUI).

[8]  L. Van Gool,et al.  Combining RGB and ToF cameras for real-time 3D hand gesture interaction , 2011, 2011 IEEE Workshop on Applications of Computer Vision (WACV).