Position Statement and Overview: Sketch Recognition at MIT

The problem with software is not that it needs a good user interface, it needs to have no user interface. Interacting with software should — ideally — feel as natural, informal, rich, and easy as working with a human assistant. One key to this lies in enabling means of interacting with software that are similarly natural, informal, rich and easy. We are making it possible for people involved in design and planning tasks to sketch, gesture, and talk about their ideas (rather than type, point, and click), and have the computer system understand their messy freehand sketches, their casual gestures, and the fragmentary utterances that are part and parcel of such interaction. A second key lies in appropriate use each of the means of interaction. Our work to date has made it clear, for example, that different means are well suited to communicating different things: Geometry is best sketched, behavior and rationale are best described in words and gestures. A third key lies in the claim that interaction will be effortless only if the listener is smart: effortless interaction and invisible interfaces must be knowledge-based. If it is to make sense of informal sketches, the listener has to understand something about the domain and something about how freehand sketches are drawn. This paper provides an overview of six current pieces of work at the MIT AI Lab on the sketch recognition part of this overall goal.