Gesture interface engine

Gesture interfaces have long been pursued in the context of portable computing and immersive environments. However, such interfaces have been difficult to realize, in part due to the lack of frameworks for their design and implementation. Analogical to a game engine which is used to build computer games, a gesture interface engine is a framework which is used to build gesture interfaces systematically, conveniently, and efficiently. Rather than using a low-level high-dimensional joint angle space, we describe and recognize handposes in a “lexical” space, in which each handpose is decomposed into elements in a finger state alphabet. A handpose is defined by finger spelling, i.e., by specifying the pose of each finger and the interrelation between any two fingers. Dynamic gestures are defined as handpose transformation, hand translation, and rotation. The alphabet and underlying grammar form a simple but expressive gesture notation system called GeLex. The implementation of the gesture interface engine is based on a Canesta laser camera that produces reliable low-resolution depth images at video rates. By decomposing and recognizing handposes as finger states, we achieve robust handpose recognition in real-time (30 frames/second). Dynamic gestures are recognized using finite state machines and explicit gesture manifold tracking. By implementing a generic gesture recognition algorithm which recognizes a large set of gestures, and a mechanism to adapt it to a specific application based on an interface description (in GeLex), the development of a gesture interface is simplified. Other functionalities provided by the gesture interface engine include: a system called “SmartCanvas” which enables a user to write and draw on a regular desk with his/her fingers; algorithms to detect and remove conflicts within a gesture set, i.e., whether two gestures used together may cause recognition errors and/or interpretion ambiguity. Also, the thesis explores the development of intuitive gesture interfaces. In an intuitive gesture interface, a user interacts with a system with natural gestures, and the intended meanings are correctly interpreted by the system using a gesture-semantic database. The goal is to minimize the learning efforts and adaptation required from a user when presented with a new gesture interface.