Toward Spontaneous Interaction with the Perceptive Workbench 1

ers mostly by using wire-based devices. Typically, the wires limit the distance of movement and inhibit freedom of orientation. In addition, most interactions are indirect. The user moves a device as an analog for the action created in the display space. We envision an untethered interface that accepts gestures directly and can accept any objects we choose as interactors. In conventional 3D interaction, the devices that track position and orientation are still usually tethered to the machine by wires. Devices, such as pinch gloves, that permit the user to experience a more natural interface often don’t perform as well. Users generally prefer simple handheld devices with buttons. Pinch gloves carry assumptions about the position of the user’s hand and fingers with respect to the tracker. Of course, users’ hands differ in size and shape, so the assumed tracker position must be recalibrated for each user. This is hardly ever done. Also, the glove interface causes subtle changes to recognized hand gestures. The result is that fine manipulations can be imprecise, and the user comes away with the feeling that the interaction is slightly off in an indeterminate way. If we can recognize gestures directly, we take into account the difference in hand sizes and shapes. An additional problem is that any device held in the hand can become awkward while gesturing. We’ve found this even with a simple pointing device, such as a stick with a few buttons (see Figure 1). Also, unless fairly skilled, a user often has to pause to identify and select buttons on the stick. With accurately tracked hands, most of this awkwardness disappears. We’re adept at pointing in almost any direction and can quickly pinch fingers, for example, without looking at them. Finally, physical objects are often natural interactors (such as phicons). However, with current systems these objects must be inserted in advance or specially prepared. We’d like the system to accept objects that we choose spontaneously for interaction. In this article we discuss methods for producing more seamless interaction between the physical and virtual environments through the Perceptive Workbench. We applied the system to an augmented reality game and a terrain navigating system. The Perceptive Workbench can reconstruct 3D virtual representations of previously unseen real-world objects placed on its surface. In addition, the Perceptive Workbench identifies and tracks such objects as they’re manipulated on the desk’s surface and allows the user to interact with the augmented environment through 2D and 3D gestures. These gestures can be made on the plane of the desk’s surface or in the 3D space above the desk. Taking its cue from the user’s actions, the Perceptive Workbench switches between these modes automatically. Computer vision controls all interaction, freeing the user from the wires of traditional sensing techniques.