A System for Hybrid Vision- and Sound-Based Interaction with Distal and Proximal Targets on Wall-Sized, High-Resolution Tiled Displays

When interacting with wall-sized, high-resolution tiled displays, users typically stand or move in front of it rather than sit at fixed locations. Using a mouse to interact can be inconvenient in this context, as it must be carried around and often requires a surface to be used. Even for devices that work in mid-air, accuracy when trying to hit small or distal targets becomes an issue. Ideally, the user should not need devices to interact with applications on the display wall. We have developed a hybrid vision- and sound-based system for device-free interaction with software running on a 7×4 tile 220-inch display wall. The system comprises three components that together enable interaction with both distal and proximal targets: (i) A camera determines the direction in which a user is pointing, allowing distal targets to be selected. The direction is determined using edge detection followed by applying the Hough transform. (ii) Using four microphones, a user double-snapping his fingers is detected and located, before the selected target is moved to the location of the snap. This is implemented using correlation and multilateration. (iii) 16 cameras detect objects (fingers, hands) in front of the display wall. The 1D positions of detected objects are then used to triangulate object positions, enabling touch-free multi-point interaction with proximal content. The system is used on the display wall in three contexts to (i) move and interact with windows from a traditional desktop interface, (ii) interact with a whiteboard-style application, and (iii) play two games.

[1]  Desney S. Tan,et al.  The large-display user experience , 2005, IEEE Computer Graphics and Applications.

[2]  Gerald D. Morrison.  A camera-based input device for large interactive displays , 2005, IEEE Computer Graphics and Applications.

[3]  Paul Wielinga,et al.  Building a 100 Mpixel graphics device for the OptIPuter , 2006, Future Gener. Comput. Syst..

[4]  George W. Fitzmaurice,et al.  A remote control interface for large displays , 2004, UIST '04.

[5]  Takeo Igarashi,et al.  Voice as sound: using non-verbal voice input for interactive control , 2001, UIST '01.

[6]  Uwe Hansmann,et al.  Pervasive Computing , 2003 .

[7]  James Scott,et al.  Audio Location: Accurate Low-Cost Location Sensing , 2005, Pervasive.

[8]  Etsuya Shibayama,et al.  The migratory cursor: accurate speech-based cursor movement by moving multiple ghost cursors using non-verbal vocalizations , 2005, Assets '05.

[9]  Daniel Vogel,et al.  Distant freehand pointing and clicking on very large, high resolution displays , 2005, UIST.

[10]  Adam Finkelstein,et al.  Building and Using A Scalable Display Wall System , 2000, IEEE Computer Graphics and Applications.

[11]  interactions Staff,et al.  CHI 2005 , 2005 .

[12]  Darren Leigh,et al.  DiamondTouch: a multi-user touch technology , 2001, UIST '01.

[13]  David A. Shamma,et al.  Visualizing live text streams using motion and temporal pooling , 2005, IEEE Computer Graphics and Applications.

[14]  Anastasia Bezerianos,et al.  The vacuum: facilitating the manipulation of distant objects , 2005, CHI.

[15]  Jefferson Y. Han.  Low-cost multi-touch sensing through frustrated total internal reflection , 2005, UIST.

[16]  Jean Rouat,et al.  Robust sound source localization using a microphone array on a mobile robot , 2003, Proceedings 2003 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2003) (Cat. No.03CH37453).

[17]  Andy Hopper,et al.  Virtual Network Computing , 1998, IEEE Internet Comput..

[18]  Richard O. Duda,et al.  Use of the Hough transformation to detect lines and curves in pictures , 1972, CACM.

[19]  John Markus Bjørndalen,et al.  Gesture-Based, Touch-Free Multi-User Gaming on Wall-Sized, High-Resolution Tiled Displays , 2008, J. Virtual Real. Broadcast..

[20]  Mary Czerwinski,et al.  Drag-and-Pop and Drag-and-Pick: Techniques for Accessing Remote Screen Content on Touch- and Pen-Operated Systems , 2003, INTERACT.