Real-time hand-tracking with a color glove

Articulated hand-tracking systems have been widely used in virtual reality but are rarely deployed in consumer applications due to their price and complexity. In this paper, we propose an easy-to-use and inexpensive system that facilitates 3-D articulated user-input using the hands. Our approach uses a single camera to track a hand wearing an ordinary cloth glove that is imprinted with a custom pattern. The pattern is designed to simplify the pose estimation problem, allowing us to employ a nearest-neighbor approach to track hands at interactive rates. We describe several proof-of-concept applications enabled by our system that we hope will provide a foundation for new interactions in modeling, animation control and augmented reality.

[1]  Michael I. Mandel,et al.  Distributed Occlusion Reasoning for Tracking with Nonparametric Belief Propagation , 2004, NIPS.

[2]  Alex Pentland,et al.  Real-Time American Sign Language Recognition Using Desk and Wearable Computer Based Video , 1998, IEEE Trans. Pattern Anal. Mach. Intell..

[3]  Marie-Paule Cani,et al.  HandNavigator: hands-on interaction for desktop virtual reality , 2008, VRST '08.

[4]  Mira Dontcheva,et al.  Layered acting for character animation , 2003, ACM Trans. Graph..

[5]  Tovi Grossman,et al.  Multi-finger gestural interaction with 3d volumetric displays , 2004, UIST '04.

[6]  Jessica K. Hodgins,et al.  Accelerometer-based user interfaces for the control of a physically simulated character , 2008, SIGGRAPH 2008.

[7]  Igor Guskov,et al.  Trackable surfaces , 2003, SCA '03.

[8]  D Kersten,et al.  Moving Cast Shadows Induce Apparent Motion in Depth , 1997, Perception.

[9]  David J. Fleet,et al.  Model-based hand tracking with texture, shading and self-occlusions , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[10]  Masood Masoodian,et al.  Bare-hand 3D gesture input to interactive systems , 2006, CHINZ '06.

[11]  David Zeltzer,et al.  A design method for “whole-hand” human-computer interaction , 1993, TOIS.

[12]  Reinhard Klein,et al.  Simultaneous 4 gestures 6 DOF real-time two-hand tracking without any markers , 2007, VRST '07.

[13]  Ryan White,et al.  Capturing and animating occluded cloth , 2007, SIGGRAPH 2007.

[14]  Taku Komura,et al.  Motion editing with data glove , 2004, ACE '04.

[15]  Ravin Balakrishnan,et al.  An interface for virtual 3D sculpting via physical proxy , 2006, GRAPHITE '06.

[16]  Marcus Magnor,et al.  Pitching a baseball: tracking high-speed motion with multi-exposure images , 2004, SIGGRAPH 2004.

[17]  Steven K. Feiner,et al.  Cross-dimensional gestural interaction techniques for hybrid immersive environments , 2005, IEEE Proceedings. VR 2005. Virtual Reality, 2005..

[18]  Steven K. Feiner,et al.  SenseShapes: using statistical geometry for object selection in a multimodal augmented reality , 2003, The Second IEEE and ACM International Symposium on Mixed and Augmented Reality, 2003. Proceedings..

[19]  Bernd Fröhlich,et al.  The two-user Responsive Workbench: support for collaboration through individual views of a shared space , 1997, SIGGRAPH.

[20]  Antonio Torralba,et al.  Small codes and large image databases for recognition , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[21]  Ying Li,et al.  Data-Driven Grasp Synthesis Using Shape Matching and Task-Based Pruning , 2007, IEEE Transactions on Visualization and Computer Graphics.

[22]  Volker Scholz,et al.  Garment Motion Capture Using Color‐Coded Patterns , 2005, Comput. Graph. Forum.

[23]  Trevor Darrell,et al.  Fast pose estimation with parameter-sensitive hashing , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[24]  Radu Horaud,et al.  The Alignment Between 3-D Data and Articulated Shapes with Bending Surfaces , 2006, ECCV.

[25]  David H. Laidlaw,et al.  Artistic Collaboration in Designing VR Visualizations , 2005, IEEE Computer Graphics and Applications.

[26]  Steven J. Gortler,et al.  A perception-based color space for illumination-invariant image processing , 2008, SIGGRAPH 2008.

[27]  Dinesh K. Pai,et al.  Interaction capture and synthesis , 2005, SIGGRAPH 2005.

[28]  Otmar Hilliges,et al.  Bringing physics to the surface , 2008, UIST '08.

[29]  Victor B. Zordan,et al.  Physically based grasping control from example , 2005, SCA '05.

[30]  KwangYun Wohn,et al.  The control of avatar motion using hand gesture , 1998, VRST '98.

[31]  Hans-Peter Seidel,et al.  FreeDrawer: a free-form sketching system on the responsive workbench , 2001, VRST '01.

[32]  Adam Finkelstein,et al.  Video puppetry: a performative interface for cutout animation , 2008, SIGGRAPH 2008.

[33]  Jun Park,et al.  LED-Glove Based Interactions in Multi-Modal Displays for Teleconferencing , 2006, 16th International Conference on Artificial Reality and Telexistence--Workshops (ICAT'06).

[34]  Brigitte Dorner,et al.  CHASING THE COLOUR GLOVE: VISUAL HAND TRACKING , 1994 .

[35]  Björn Stenger,et al.  Model-based hand tracking using a hierarchical Bayesian filter , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[36]  George Kollios,et al.  BoostMap: A method for efficient approximate similarity rankings , 2004, CVPR 2004.

[37]  Stan Sclaroff,et al.  Estimating 3D hand pose from a cluttered image , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..