Multi-camera Finger Tracking and 3D Trajectory Reconstruction for HCI Studies

Three-dimensional human-computer interaction has the potential to form the next generation of user interfaces and to replace the current 2D touch displays. To study and to develop such user interfaces, it is essential to be able to measure how a human behaves while interacting with them. In practice, this can be achieved by accurately measuring hand movements in 3D by using a camera-based system and computer vision. In this work, a framework for multi-camera finger movement measurements in 3D is proposed. This includes comprehensive evaluation of state-of-the-art object trackers to select the most appropriate one to track fast gestures such as pointing actions. Moreover, the needed trajectory post-processing and 3D trajectory reconstruction methods are proposed. The developed framework was successfully evaluated in the application where 3D touch screen usability is studied with 3D stimuli. The most sustainable performance was achieved by the Structuralist Cognitive model for visual Tracking tracker complemented with the LOESS smoothing.

[1]  Andrew Zisserman,et al.  Multiple View Geometry in Computer Vision (2nd ed) , 2003 .

[2]  James L. Lyons,et al.  Goal-directed aiming: two components but multiple processes. , 2010, Psychological bulletin.

[3]  Melvyn A. Goodale,et al.  The role of binocular vision in prehension: a kinematic analysis , 1992, Vision Research.

[4]  David Zhang,et al.  Fast Visual Tracking via Dense Spatio-temporal Context Learning , 2014, ECCV.

[5]  Bernhard P. Wrobel,et al.  Multiple View Geometry in Computer Vision , 2001 .

[6]  Zhenyu He,et al.  The Visual Object Tracking VOT2016 Challenge Results , 2016, ECCV Workshops.

[7]  W. Cleveland,et al.  Locally Weighted Regression: An Approach to Regression Analysis by Local Fitting , 1988 .

[8]  Jari Takatalo,et al.  High-Speed Hand Tracking for Studying Human-Computer Interaction , 2015, SCIA.

[9]  Rui Caseiro,et al.  Exploiting the Circulant Structure of Tracking-by-Detection with Kernels , 2012, ECCV.

[10]  A. Murat Tekalp,et al.  Performance measures for video object segmentation and tracking , 2003, IEEE Transactions on Image Processing.

[11]  Carlo Tomasi,et al.  Good features to track , 1994, 1994 Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.

[12]  Jane Yung-jen Hsu,et al.  Touching the void: direct-touch interaction for intangible displays , 2010, CHI.

[13]  Robert Laganière,et al.  Scalable Kernel Correlation Filter with Sparse Feature Integration , 2015, 2015 IEEE International Conference on Computer Vision Workshop (ICCVW).

[14]  Dorin Comaniciu,et al.  Real-time tracking of non-rigid objects using mean shift , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[15]  Mircea Nicolescu,et al.  Vision-based hand pose estimation: A review , 2007, Comput. Vis. Image Underst..

[16]  Cordelia Schmid,et al.  Learning Color Names for Real-World Applications , 2009, IEEE Transactions on Image Processing.

[17]  Rama Chellappa,et al.  In Situ Evaluation of Tracking Algorithms Using Time Reversed Chains , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[18]  Toni Kuronen.  Post-processing and analysis of tracked hand trajectories , 2014 .

[19]  Kevin Nickels,et al.  Estimating uncertainty in SSD-based feature tracking , 2002, Image Vis. Comput..

[20]  Klaus H. Hinrichs,et al.  Evaluation of depth perception for touch interaction with stereoscopic rendered objects , 2012, ITS.

[21]  Ming-Hsuan Yang,et al.  Incremental Learning for Robust Visual Tracking , 2008, International Journal of Computer Vision.

[22]  Jiri Matas,et al.  Robust scale-adaptive mean-shift for tracking , 2013, Pattern Recognit. Lett..

[23]  Tuomas Eerola,et al.  Comparison of General Object Trackers for Hand Tracking in High-Speed Videos , 2014, 2014 22nd International Conference on Pattern Recognition.

[24]  Alexander Toet,et al.  Visual comfort of binocular and 3D displays , 2004 .

[25]  Wijnand A. IJsselsteijn,et al.  Stereoscopic displays in medical domains: a review of perception and performance effects , 2009, Electronic Imaging.

[26]  Thomas Mauthner,et al.  In defense of color-based model-free tracking , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[27]  Rui Caseiro,et al.  High-Speed Tracking with Kernelized Correlation Filters , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[28]  Jiri Matas,et al.  Forward-Backward Error: Automatic Detection of Tracking Failures , 2010, 2010 20th International Conference on Pattern Recognition.

[29]  Yiannis Demiris,et al.  Visual Tracking Using Attention-Modulated Disintegration and Integration , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[30]  Luca Bertinetto,et al.  Staple: Complementary Learners for Real-Time Tracking , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).