3D natural hand interaction for AR applications

In this paper we describe a computer vision-based 3D hand tracking system for a multimodal augmented reality (AR) interface. We have developed a 3D vision-based natural hand interaction method. This consists of four steps: (1) skin colour segmentation, (2) feature point finding, (3) hand direction calculation, and (4) simple collision detection based on a short finger ray for interaction between the userpsilas hand and augmented objects. The resulting fingertip tracking accuracy varied from 3 mm to 20 mm depending on the distance between the userpsilas hand and the stereo camera. We describe how this hand tracking is applied in three AR applications which merge gesture and speech input.

[1]  Heedong Ko,et al.  An Evaluation of an Augmented Reality Multimodal Interface Using Speech and Paddle Gestures , 2006, ICAT.

[2]  Tobias Höllerer,et al.  Handy AR: Markerless Inspection of Augmented Reality Objects Using Fingertip Tracking , 2007, 2007 11th IEEE International Symposium on Wearable Computers.

[3]  Daniel Vogel,et al.  Distant freehand pointing and clicking on very large, high resolution displays , 2005, UIST.

[4]  J. Edward Swan,et al.  Survey of User-Based Experimentation in Augmented Reality , 2005 .

[5]  Bruce H. Thomas,et al.  Using ARToolKit for 3D hand position tracking in mobile outdoor environments , 2002, The First IEEE International Workshop Agumented Reality Toolkit,.

[6]  Heedong Ko,et al.  "Move the couch where?" : developing an augmented reality multimodal interface , 2006, 2006 IEEE/ACM International Symposium on Mixed and Augmented Reality.

[7]  Gunther Heidemann,et al.  Multimodal interaction in an augmented reality scenario , 2004, ICMI '04.

[8]  Tobias Höllerer,et al.  Vision-based interfaces for mobility , 2004, The First Annual International Conference on Mobile and Ubiquitous Systems: Networking and Services, 2004. MOBIQUITOUS 2004..

[9]  Ivan Poupyrev,et al.  Virtual object manipulation on a table-top AR environment , 2000, Proceedings IEEE and ACM International Symposium on Augmented Reality (ISAR 2000).

[10]  Gunilla Borgefors,et al.  Distance transformations in digital images , 1986, Comput. Vis. Graph. Image Process..

[11]  Hirokazu Kato,et al.  Marker tracking and HMD calibration for a video-based augmented reality conferencing system , 1999, Proceedings 2nd IEEE and ACM International Workshop on Augmented Reality (IWAR'99).

[12]  Nicoletta Adamo-Villani,et al.  ARpm: an augmented reality interface for polygonal modeling , 2005, Fourth IEEE and ACM International Symposium on Mixed and Augmented Reality (ISMAR'05).

[13]  Ronald Azuma,et al.  Recent Advances in Augmented Reality , 2001, IEEE Computer Graphics and Applications.

[14]  Alexander H. Waibel,et al.  Segmenting hands of arbitrary color , 2000, Proceedings Fourth IEEE International Conference on Automatic Face and Gesture Recognition (Cat. No. PR00580).

[15]  Herbert Freeman,et al.  Computer Processing of Line-Drawing Images , 1974, CSUR.

[16]  Mark Fiala,et al.  ARTag, a fiducial marker system using digital techniques , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).