Building a gesture based information terminal

In this thesis a Multi-Pointer Virtual Touchscreen system is developed, which allows the application of pointing gestures in a standard GUI interface. Unlike in most related projects, an acoustic tap detection method supports the optical tracking. The system is designed as an information terminal for public places, though other applications are possible. Its uses a projected beamer screen as output. Users can control several pointers, similar to mouse pointers in a standard desktop interface. Selection of objects and drag-and-drop actions can be done by tapping the table surface with the finger tip. The optical tracking system derives one pointer position per hand. A novel Multi-Pointer X-Server (MPX) is utilized and configured to handle these coordinates similar to normal mouse input. The new architecture provides the ability to display several mouse cursors and supports Multi-Pointer aware applications. These applications handle the input independently and allow simultaneous actions. Aside from that, standard applications can be operated in conventional manner. For detection of surface touches, a tangible acoustic interface is applied. Tap locations are distinguished implementing Time Difference of Arrival (TDOA) estimation. The mathematical basis for this approach is the Generalized Cross Correlation with Phase Transform (GCC-PHAT). Employing only a stereo audio input allows differentiation of tap locations, though archived accuracy is still limited.