What: While sketches are widely used in the early stages of design, not all aspects of a design are easily sketched; some are more easily described verbally. We will create a more natural interaction for the user by being able to handle both modalities, allowing users to sketch and talk simultaneously, while also enabling the computer to better understand the user’s intentions. Why: Ourprevioussystem, ASSIST[2], recognizesmechanicalcomponents(e.g., springs, pulleys, axles, etc.) that users sketch in a natural fashion with a pen-like input device. ASSIST displays a “cleaned up” version of the user’s sketch and interfaces with a physics simulation tool to show an animated version of the sketch. Newton’s Cradle (see Figure 1) is a set of pendulums consisting of a row of metal balls on strings. When you pull back and release a number of balls on one end, after a nearly elastic collision, the same number of balls will move outward on the other end. Although this appears to be easily sketched, it is nearly impossible to draw so that it operates properly. The metal balls must just touch each other and the pendulums must be identical. If the user could simply say that “there are five identical, evenly spaced and touching pendulums,” the device would be easy to create.
[1]
Marilyn A. Walker,et al.
MATCH: An Architecture for Multimodal Dialogue Systems
,
2002,
ACL.
[2]
Sharon L. Oviatt,et al.
Designing the User Interface for Multimodal Speech and Pen-Based Gesture Applications: State-of-the-Art Systems and Future Research Directions
,
2000,
Hum. Comput. Interact..
[3]
Michael Oltmans,et al.
Understanding Naturally Conveyed Explanations of Device Behavior
,
2000
.
[4]
Joseph Polifroni,et al.
Recognition confidence scoring and its use in speech understanding systems
,
2002,
Comput. Speech Lang..
[5]
Randall Davis,et al.
Speech and sketching for multimodal design
,
2004,
IUI '04.