Drawing on the Back of an Envelope: a framework for interacting with application programs by freehand drawing

Abstract The Back of an Envelope project demonstrates how a calligraphic interface (one that employs a pen or stylus to input freehand drawing marks) can be used in a wide variety of domains, from databases to simulation programs, to 3D modeling, from mostly symbolic diagrams to freeform sketches. The wide variety of drawing types and domains calls for a diverse range of approaches. We describe some of the functionality of our systems, including contextual recognition of symbols and configurations and emergent shape recognition, and some of the calligraphic interfaces we’ve built.

[1]  Eric and Gross Mark Schweikhardt.  Digital Clay: Deriving Digital Models from Freehand Sketches , 1998 .

[2]  Leysia Palen,et al.  “I'll get that off the audio”: a case study of salvaging multimedia meeting records , 1997, CHI.

[3]  Eric J. Golin,et al.  The specification of visual language syntax , 1990, J. Vis. Lang. Comput..

[4]  Ken Perlin,et al.  Pad: an alternative approach to the computer interface , 1993, SIGGRAPH.

[5]  Napkin Gross,et al.  Demonstrating the Electronic Cocktail , 1996 .

[6]  Martin Odersky,et al.  Building visual language parsers , 1991, CHI '91.

[7]  Rui Zhao,et al.  Incremental recognition in gesture-based and syntax-directed diagram editors , 1993, INTERCHI.

[8]  William Newman,et al.  A desk supporting computer-based interaction with paper documents , 1992, CHI.

[9]  Peter van Sommers,et al.  Drawing and Cognition: Descriptive and Experimental Studies of Graphic Production Processes , 1984 .

[10]  Brian V. Funt,et al.  Problem-Solving with Diagrammatic Representations , 1980, Artif. Intell..

[11]  Ellen Yi-Luen Do,et al.  The right tool at the right time: investigation of freehand drawing as an interface to knowledge-based design tools , 1998 .

[12]  Kim Marriott.  Constraint multiset grammars , 1994, Proceedings of 1994 IEEE Symposium on Visual Languages.

[13]  Gregory D. Abowd,et al.  Teaching and learning as multimedia authoring: the classroom 2000 project , 1997, MULTIMEDIA '96.

[14]  Ioannis A. Kakadiaris,et al.  Understanding diagrams in technical documents , 1992, Computer.

[15]  Mark D. Gross,et al.  The Electronic Cocktail Napkin—a computational environment for working with design diagrams , 1996 .

[16]  Axel Kramer,et al.  Translucent patches—dissolving windows , 1994, UIST '94.

[17]  Edward Robbins,et al.  Why Architects Draw , 1997 .

[18]  Wendy E. Mackay,et al.  Ariel: augmenting paper engineering drawings , 1995, CHI 95 Conference Companion.

[19]  Dean Rubine,et al.  Specifying gestures by example , 1991, SIGGRAPH.

[20]  M. Gross,et al.  Digital clay: deriving digital models from freehand sketches , 2000 .

[21]  Herbert A. Simon,et al.  Why a Diagram is (Sometimes) Worth Ten Thousand Words , 1987 .

[22]  Pierre David Wellner,et al.  Interacting with paper on the DigitalDesk , 1993, CACM.

[23]  V. Goel.  Sketches of thought , 1995 .

[24]  Yin Yin Wong.  Rough and ready prototypes: lessons from graphic design , 1992, CHI '92.

[25]  R. A. Kirsch,et al.  Experiments in processing pictorial information with a digital computer , 1899, IRE-ACM-AIEE '57 (Eastern).

[26]  James A. Landay,et al.  Interactive sketching for the early stages of user interface design , 1995, CHI '95.

[27]  Takeo Igarashi,et al.  Flatland: new dimensions in office whiteboards , 1999, CHI '99.

[28]  Michael Trinder,et al.  The Computer’s Role in Sketch Design: A Transparent Sketching Medium , 1999 .

[29]  Thomas P. Moran,et al.  Implicit structure for pen-based systems within a freeform interaction paradigm , 1995, CHI '95.

[30]  Bernd Meyer,et al.  Pictures depicting pictures on the specification of visual languages by visual grammars , 1992, Proceedings IEEE Workshop on Visual Languages.

[31]  Gershon Elber,et al.  Sketching as a solid modeling tool , 1995, SMA '95.

[32]  Thomas F. Stahovich,et al.  Generating Multiple New Designs from a Sketch , 1996, Artif. Intell..

[33]  Nicholas Negroponte,et al.  Recent advances in sketch recognition , 1973, AFIPS National Computer Conference.

[34]  Thomas P. Moran,et al.  Spatial interpretation of domain objects integrated into a freeform electronic whiteboard , 1998, UIST '98.

[35]  Ellen Yi-Luen Do,et al.  Space Pen Annotation and sketching on 3D models on the Internet , 2001 .

[36]  Fred Lakin.  Visual Grammars for Visual Languages , 1987, AAAI.

[37]  Charles M. Eastman,et al.  Computers in Building: Proceedings of the CAAD Futures '99 Conference , 1999 .

[38]  Christopher F. Herot.  Graphical input through machine recognition of sketches , 1976, SIGGRAPH '76.

[39]  Wendy E. Mackay,et al.  Augmenting reality: adding computational dimensions to paper , 1993, CACM.

[40]  N. Hari Narayanan,et al.  Diagrammatic Reasoning: Cognitive and Computational Perspectives , 1995 .

[41]  Ralph R. Martin,et al.  Creating solid models from single 2D sketches , 1995, Symposium on Solid Modeling and Applications.

[42]  Hiroshi Ishii,et al.  ClearBoard: a seamless medium for shared drawing and conversation with eye contact , 1992, CHI.

[43]  Ephraim P. Glinert,et al.  Pict: An Interactive Graphical Programming Environment , 1984, Computer.

[44]  Bill N. Schilit,et al.  Beyond paper: supporting active reading with free form digital ink annotations , 1998, CHI.

[45]  Herbert A. Simon,et al.  Why a diagram is (sometimes) worth 10, 000 word , 1987 .

[46]  Marvin Minsky,et al.  Semantic Information Processing , 1968 .

[47]  T. G. Evans.  A program for the solution of a class of geometric-analogy intelligence-test questions , 1964 .

[48]  Carl Gutwin,et al.  The usability of transparent overview layers , 1998, CHI Conference Summary.

[49]  M. J. Enenhofer.  Spatial-Query-by-Sketch , 1996, Proceedings 1996 IEEE Symposium on Visual Languages.

[50]  M. B. Clowes,et al.  On Seeing Things , 1971, Artif. Intell..

[51]  Mark D. Gross,et al.  Stretch-A-Sketch: a dynamic diagrammer , 1994, Proceedings of 1994 IEEE Symposium on Visual Languages.

[52]  Jonathan Stephen Fish,et al.  Amplifying the Mind’s Eye: Sketching and Visual Cognition , 1990 .

[53]  Clive Frankish,et al.  Recognition accuracy and user acceptance of pen interfaces , 1995, CHI '95.

[54]  Eric J. Golin.  A method for the specification and parsing of visual languages , 1991 .

[55]  Lynn Wilcox,et al.  A dynamic grouping technique for ink and audio notes , 1998, UIST '98.

[56]  David Craig,et al.  The importance of drawing in the mechanical design process , 1990, Comput. Graph..

[57]  Sharon L. Oviatt,et al.  Perceptual user interfaces: multimodal interfaces that process what comes naturally , 2000, CACM.

[58]  Takayuki Dan Kimura,et al.  A comparison study of the pen and the mouse in editing graphic diagrams , 1993, Proceedings 1993 IEEE Symposium on Visual Languages.

[59]  Philip R. Cohen,et al.  MULTIMODAL INTERFACES THAT PROCESS WHAT COMES NATURALLY , 2000 .

[60]  Ivan E. Sutherland,et al.  Sketchpad a Man-Machine Graphical Communication System , 1899, Outstanding Dissertations in the Computer Sciences.