Robust real-time upper body limb detection and tracking

We describe an efficient and robust system to detect and track the limbs of a human. Of special consideration in the design of this system are real-time and robustness issues. We thus utilize a detection/tracking scheme in which we detect the face and limbs of a user and then track the forearms of the found limbs. Detection occurs by first finding the face of a user. The location and color information from the face can then be used to find limbs. As skin color is a key visual feature in this system, we continuously search for faces and use them to update skin color information. Along with edge information, this is used in the subsequent forearm tracking. Robustness is implicit in this design, as the system automatically re-detects a limbs when its corresponding forearms is lost. This design is also conducive to real-time processing because while detection of the limbs can take up to seconds, tracking is on the order of milliseconds. Thus reasonable frame rates can be achieved with a short latency. Also, in this system we make use of multiple 2D limb tracking models to enhance tracking of the underlying 3D structure. This includes models for lateral forearm views (waving) as well as for pointing gestures. Experiments on test sequences demonstrate the efficacy of this approach.