Perceptually-based language to simplify sketch recognition user interface development

Diagrammatic sketching is a natural modality of human-computer interaction that can be used for a variety of tasks, for example, conceptual design. Sketch recognition systems are currently being developed for many domains. However, they require signal-processing expertise if they are to handle the intricacies of each domain, and they are time-consuming to build. Our goal is to enable user interface designers and domain experts who may not have expertise in sketch recognition to be able to build these sketch systems. We created and implemented a new framework (FLUID - facilitating user interface development) in which developers can specify a domain description indicating how domain shapes are to be recognized, displayed, and edited. This description is then automatically transformed into a sketch recognition user interface for that domain. LADDER, a language using a perceptual vocabulary based on Gestalt principles, was developed to describe how to recognize, display, and edit domain shapes. A translator and a customizable recognition system (GUILD - a generator of user interfaces using ladder descriptions) are combined with a domain description to automatically create a domain specific recognition system. With this new technology, by writing a domain description, developers are able to create a new sketch interface for a domain, greatly reducing the time and expertise for the task. Continuing in pursuit of our goal to facilitate UI development, we noted that (1) human generated descriptions contained syntactic and conceptual errors, and that (2) it is more natural for a user to specify a shape by drawing it than by editing text. However, computer generated descriptions from a single drawn example are also flawed, as one cannot express all allowable variations in a single example. In response, we created a modification of the traditional model of active learning in which the system selectively generates its own near-miss examples and uses the human teacher as a source of labels. System generated near-misses offer a number of advantages. Human generated examples are tedious to create and may not expose problems in the current concept. It seems most effective for the near-miss examples to be generated by whichever learning participant (teacher or student) knows better where the deficiencies lie; this will allow the concepts to be more quickly and effectively refined. When working in a closed domain such as this one, the computer learner knows exactly which conceptual uncertainties remain, and which hypotheses need to be tested and confirmed. The system uses these labeled examples to automatically build a LADDER shape description, using a modification of the version spaces algorithm that handles interrelated constraints, and which also has the ability to learn negative and disjunctive constraints.

[1]  Philip R. Cohen,et al.  QuickSet: multimodal interaction for distributed applications , 1997, MULTIMEDIA '97.

[2]  Federico Bergenti,et al.  Agent-oriented Software Construction with UML , 2001 .

[3]  R. Krauss.  Why Do We Gesture When We Speak? , 1998 .

[4]  Sinan Si Alhir.  UML in a nutshell - a desktop quick reference , 1998 .

[5]  Christine J. Alvarado,et al.  A natural sketching environment : bringing the computer into early stages of mechanical design , 2000 .

[6]  Ramesh C. Jain,et al.  A survey on the use of pattern recognition methods for abstraction, indexing and retrieval of images and video , 2002, Pattern Recognit..

[7]  Michael Oltmans,et al.  Understanding Naturally Conveyed Explanations of Device Behavior , 2000 .

[8]  James A. Landay,et al.  Interactive sketching for the early stages of user interface design , 1995, CHI '95.

[9]  Michael Johnston,et al.  Balancing data-driven and rule-based approaches in the context of a multimodal conversational system , 2003, 2003 IEEE Workshop on Automatic Speech Recognition and Understanding (IEEE Cat. No.03EX721).

[10]  Allan D. Jepson,et al.  Qualitative probabilities for image interpretation , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[11]  A. Richard Newton,et al.  Recognition and beautification of multi-stroke symbols in digital ink , 2005, Comput. Graph..

[12]  James A. Landay,et al.  Sketching Interfaces: Toward More Human Interface Design , 2001, Computer.

[13]  Robin Jeffries,et al.  User interface evaluation in the real world: a comparison of four techniques , 1991, CHI.

[14]  Thomas F. Stahovich,et al.  SketchIT: a Sketch Interpretation Tool for Conceptual Mechanical Design , 1996 .

[15]  John Langford,et al.  Agnostic active learning , 2006, J. Comput. Syst. Sci..

[16]  James A. Landay,et al.  Visual similarity of pen gestures , 2000, CHI.

[17]  Anderson Michael,et al.  AAAI Fall Symposium , 2005 .

[18]  Nicholas E. Matsakis.  Recognition of Handwritten Mathematical Expressions , 1999 .

[19]  James F. Allen,et al.  Toward Conversational Human-Computer Interaction , 2001, AI Mag..

[20]  Robert C. Miller,et al.  Off-Line Sketch Interpretation , 2004, AAAI Technical Report.

[21]  Robert L. Mason,et al.  Some practical aspects of covariance estimation , 1985, Pattern Recognit. Lett..

[22]  Mark W. Newman,et al.  Informal PUIs: No Recognition Required , 2002 .

[23]  Christine Alvarado.  Dynamically Constructed Bayesian Networks for Sketch Understanding , 2003 .

[24]  S. Goldin-Meadow,et al.  What the teacher's hands tell the student's mind about math. , 1999 .

[25]  Robert E. Horn,et al.  Visual Language: Global Communication for the 21st Century , 1998 .

[26]  Tracy Hammond.  Natural Sketch Recognition in UML Class Diagrams , 2001 .

[27]  E. J. Friedman-hill,et al.  Jess, the Java expert system shell , 1997 .

[28]  Randall Davis,et al.  Speech and sketching for multimodal design , 2004, IUI '04.

[29]  Ivar Jacobson,et al.  The Unified Modeling Language User Guide , 1998, J. Database Manag..

[30]  Mark D. Gross,et al.  Recognizing and interpreting diagrams in design , 1994, AVI '94.

[31]  Elaine Cohen,et al.  Real time spline curves from interactively sketched data , 1990, I3D '90.

[32]  Dana Angluin.  Queries revisited , 2004, Theor. Comput. Sci..

[33]  Thomas G. Dietterich.  What is machine learning? , 2020, Archives of Disease in Childhood.

[34]  Barbara Tversky,et al.  What do Sketches Say about Thinking , 2002 .

[35]  James A. Landay,et al.  Sketch-based user interfaces for collaborative object-oriented modeling , 1999 .

[36]  Song Han,et al.  3DSketch: modeling by digitizing with a smart 3D pen , 1997, MULTIMEDIA '97.

[37]  Clark F. Olson.  Probabilistic Indexing for Object Recognition , 1995, IEEE Trans. Pattern Anal. Mach. Intell..

[38]  Andrew Gelsey.  Automated Reasoning about Machine Geometry and Kinematics , 1989 .

[39]  Tracy Hammond,et al.  Gender-based underrepresentation in computer science and related disciplines , 2002, 32nd Annual Frontiers in Education.

[40]  Randall Davis,et al.  A New Approach to Early Sketch Processing , 2004, AAAI Technical Report.

[41]  Jan-Olof Eklundh,et al.  Shape Representation by Multiscale Contour Approximation , 1991, IEEE Trans. Pattern Anal. Mach. Intell..

[42]  James R. Gattiker,et al.  A System for Interpretation of Line Drawings , 1990, IEEE Trans. Pattern Anal. Mach. Intell..

[43]  James A. Landay,et al.  "Those look similar!" issues in automating gesture design advice , 2001, PUI '01.

[44]  Randall Davis,et al.  SHADY: A Shape Description Debugger for Use in Sketch Recognition , 2004, AAAI Technical Report.

[45]  Edward Filisko,et al.  Virtual Modality: a Framework for Testing and Building Multimodal Applications , 2004, Workshop On Spoken Language Understanding For Conversational Systems And Higher Level Linguistic Information For Speech Processing.

[46]  Sharon L. Oviatt,et al.  Designing the User Interface for Multimodal Speech and Pen-Based Gesture Applications: State-of-the-Art Systems and Future Research Directions , 2000, Hum. Comput. Interact..

[47]  Kenneth D. Forbus,et al.  A Cognitive Approach to Sketch Understanding , 2002 .

[48]  Mark D. Gross,et al.  The Electronic Cocktail Napkin—a computational environment for working with design diagrams , 1996 .

[49]  Sharon L. Oviatt,et al.  The efficiency of multimodal interaction: a case study , 1998, ICSLP.

[51]  Thomas Baudel,et al.  A mark-based interaction paradigm for free-hand drawing , 1994, UIST '94.

[52]  Kenneth D. Forbus,et al.  Sketching for military courses of action diagrams , 2003, IUI '03.

[53]  Howard E. Shrobe,et al.  Using semantic networks for knowledge representation in an intelligent environment , 2003, Proceedings of the First IEEE International Conference on Pervasive Computing and Communications, 2003. (PerCom 2003)..

[54]  Mark W. Newman,et al.  DENIM: finding a tighter fit between tools and practice for Web site design , 2000, CHI.

[55]  James Gips,et al.  Computer Implementation of Shape Grammars , 1999 .

[56]  M. GROSS,et al.  Using diagrams to access a case library of architectural designs , 1996 .

[57]  Ellen Yi-Luen Do,et al.  Ambiguous intentions: a paper-like interface for creative design , 1996, UIST '96.

[58]  Rakesh Mohan,et al.  Multidimensional indexing for recognizing visual shapes , 1991, CVPR 1991.

[59]  James V. Mahoney,et al.  Handling ambiguity in constraint-based recognition of stick figure sketches , 2001, IS&T/SPIE Electronic Imaging.

[60]  Ephraim P. Glinert,et al.  Online parsing of visual languages using adjacency grammars , 1995, Proceedings of Symposium on Visual Languages.

[61]  A. Richard Newton,et al.  Robust sketched symbol fragmentation using templates , 2004, IUI '04.

[62]  James Arvo,et al.  Fluid sketches: continuous recognition and morphing of simple hand-drawn shapes , 2000, UIST '00.

[63]  Ellen Yi-Luen Do.  VR Sketchpad. Create Instant 3D Worlds by Sketching on a Transparent Window , 2001 .

[64]  Joaquim A. Jorge,et al.  Visual languages for sketching documents , 2000, Proceeding 2000 IEEE International Symposium on Visual Languages.

[65]  tephen E. Palmer.  The effects of contextual scenes on the identification of objects , 1975, Memory & cognition.

[66]  Krzysztof Z. Gajos,et al.  Ligature: Gesture-Based Configuration of the E21 Intelligent Environment , 2001 .

[67]  Patrick Henry Winston,et al.  Learning structural descriptions from examples , 1970 .

[68]  Levent Burak Kara,et al.  Combining geometry and domain knowledge to interpret hand-drawn diagrams , 2005, Comput. Graph..

[69]  Lin Wu,et al.  MeetingManager: A Collaborative Tool in the Intelligent Room , 2007 .

[70]  Manoj D. Muzumdar.  ICEMENDR : intelligent capture environment for mechanical engineering drawing , 1999 .

[71]  Pericles Loucopoulos,et al.  A generic model for reflective design , 2000, TSEM.

[72]  Jacob Eisenstein,et al.  Gestural Cues for Sentence Segmentation , 2005 .

[73]  David G. Lowe,et al.  Perceptual Organization and Visual Recognition , 2012 .

[74]  Gregory D. Abowd,et al.  Providing integrated toolkit-level support for ambiguity in recognition-based interfaces , 2000, CHI.

[75]  Ellen Yi-Luen Do,et al.  Drawing as a Means to Design Reasoning , 1996 .

[76]  Bernhard Schölkopf,et al.  Introduction to Semi-Supervised Learning , 2006, Semi-Supervised Learning.

[77]  Takeo Kanade,et al.  Automatic generation of object recognition programs , 1988, Proc. IEEE.

[78]  Tracy Hammond,et al.  Automatically Generating Sketch Interfaces from Shape Descriptions , 2004 .

[79]  Manuel J. Fonseca,et al.  Sketching User Interfaces with Visual Patterns , .

[80]  Michael H. Coen,et al.  Meeting the Computational Needs of Intelligent Environments: The Metaglue System , 2000 .

[81]  Victor Zue,et al.  Conversational interfaces: advances and challenges , 1997, Proceedings of the IEEE.

[82]  R. Davis,et al.  Perceptually Based Learning of Shape Descriptions , 2002 .

[83]  James V. Mahoney,et al.  Interpreting Sloppy Stick Figures by Graph Rectification and Constraint-Based Matching , 2001, GREC.

[84]  Elliotte Rusty Harold,et al.  XML in a nutshell - a desktop quick reference: covers XML 1.1. and XInclude (3. ed.) , 2000 .

[85]  Ivar Jacobson,et al.  Unified Modeling Language , 2020, Definitions.

[86]  L. Kara,et al.  Recognizing Multi-Stroke Symbols , 2002 .

[87]  Marti A. Hearst.  Sketching intelligent systems , 1998 .

[88]  Christopher F. Herot.  Graphical input through machine recognition of sketches , 1976, SIGGRAPH '76.

[89]  Christian Heide Damm,et al.  Tool support for cooperative object-oriented design: gesture based modelling on an electronic whiteboard , 2000, CHI.

[90]  Satoshi Matsuoka,et al.  Teddy: A Sketching Interface for 3D Freeform Design , 1999, SIGGRAPH Courses.

[91]  Ann Kipling Brown,et al.  Dance notation for beginners , 1984 .

[92]  Krzysztof Z. Gajos,et al.  An Agent-Based System for Capturing and Indexing Software Design Meetings , 2002 .

[93]  David A. Cohn,et al.  Improving generalization with active learning , 1994, Machine Learning.

[94]  Rob Miller,et al.  The Amulet Environment: New Models for Effective User Interface Software Development , 1997, IEEE Trans. Software Eng..

[95]  Randall Davis,et al.  A Domain Description Language for Sketch Recognition , 2002 .

[96]  J. Bruner.  The act of discovery. , 1961 .

[97]  Robert P. Futrelle,et al.  Constraint based vectorization , 1989, ICS '89.

[98]  David J. Fleet,et al.  Perceptual Organization as a Foundation for Intelfigent Sketch Editing , 2002 .

[99]  Kenneth D. Forbus,et al.  Towards a computational model of sketching , 2001, IUI '01.

[100]  Ellen Yi-Luen Do,et al.  Demonstrating the electronic cocktail napkin: a paper-like interface for early design , 1996, CHI Conference Companion.

[101]  Aaron Adler.  Segmentation and Alignment of Speech and Sketching in a Design Environment , 2003 .

[102]  John M. Carroll,et al.  Design rationale: concepts, techniques, and use , 1996 .

[103]  C. Bonwell,et al.  Active learning : creating excitement in the classroom , 1991 .

[104]  Joaquim A. Jorge,et al.  CALI: An Online Scribble Recognizer for Calligraphic Interfaces , 2002 .

[105]  Gérard G. Medioni,et al.  Structural Indexing: Efficient 3-D Object Recognition , 1992, IEEE Trans. Pattern Anal. Mach. Intell..

[106]  Robin L. Kullberg.  Mark your calendar!: learning personalized annotation from integrated sketch and speech , 1995, CHI 95 Conference Companion.

[107]  Joachim M. Buhmann,et al.  Distortion Invariant Object Recognition in the Dynamic Link Architecture , 1993, IEEE Trans. Computers.

[108]  Takayuki Dan Kimura,et al.  Recognizing multistroke geometric shapes: an experimental evaluation , 1993, UIST '93.

[109]  Suk I. Yoo,et al.  Image retrieval using topological structure of user sketch , 1998, SMC'98 Conference Proceedings. 1998 IEEE International Conference on Systems, Man, and Cybernetics (Cat. No.98CH36218).

[110]  Michael I. Jordan,et al.  Forward Models: Supervised Learning with a Distal Teacher , 1992, Cogn. Sci..

[111]  Randall Davis,et al.  LADDER, a sketching language for user interface developers , 2005, Comput. Graph..

[112]  Jeffrey D. Karpicke,et al.  The Power of Testing Memory Basic Research and Implications for Educational Practice , 2006 .

[113]  Ajay Kulkarni,et al.  Building Agent-Based Intelligent Workspaces , 2002, International Conference on Internet Computing.

[114]  Levent Burak Kara,et al.  An Image-Based Trainable Symbol Recognizer for Sketch-Based Interfaces , 2004, AAAI Technical Report.

[115]  Charles W. Anderson,et al.  Efficient indexing for object recognition using large networks , 1997, Proceedings of International Conference on Neural Networks (ICNN'97).

[116]  Bernhard Bauer,et al.  Extending UML for agents , 2000 .

[117]  James A. Landay,et al.  Quill: a gesture design tool for pen-based user interfaces , 2001 .

[118]  Edward Lank.  A retargetable framework for interactive diagram recognition , 2003, Seventh International Conference on Document Analysis and Recognition, 2003. Proceedings..

[119]  Jacob Eisenstein,et al.  Visual and linguistic information in gesture classification , 2006 .

[120]  Steven K. Feiner,et al.  Introduction to Computer Graphics Macintosh Version Software of SRGP and SPHIGS Software , 1994 .

[121]  D. f. l. Jenkins,et al.  Applying constraints to enforce users' intentions in free-hand 2-D sketches , 1992 .

[122]  Genny Tortora,et al.  Automatic Generation of Visual Programming Environments , 1995, Computer.

[123]  Eric Lecolinet.  Designing GUIs by sketch drawing and visual programming , 1998, AVI '98.

[124]  Ivan E. Sutherland,et al.  Sketchpad: a man-machine graphical communication system , 1899, AFIPS '63 (Spring).

[125]  Dean Rubine,et al.  Specifying gestures by example , 1991, SIGGRAPH.

[126]  Ellen Yi-Luen Do,et al.  Functional and Formal Reasoning in Architectural Sketches , 2002 .

[127]  Matti Kääriäinen,et al.  Active Learning in the Non-realizable Case , 2006, ALT.

[128]  Linda G. Shapiro,et al.  3D Object Recognition and Pose with Relational Indexing , 2000, Comput. Vis. Image Underst..

[129]  Ivan Marsic,et al.  A framework for rapid development of multimodal interfaces , 2003, ICMI '03.

[130]  Randall Davis,et al.  Tahuti: a geometrical sketch recognition system for UML class diagrams , 2006, SIGGRAPH Courses.

[131]  N. D. Belajcic.  Computer Implementation of Shape Grammars , 1988 .

[132]  Christine Alvarado,et al.  Multi-domain sketch understanding , 2007, SIGGRAPH Courses.

[133]  Jason Hong,et al.  Sketch Recognizers from the End-User's, the Designer's, and the Programmer's Perspective , 2002 .

[134]  Toshikazu Kato,et al.  A sketch retrieval method for full color image database-query by visual example , 1992, [1992] Proceedings. 11th IAPR International Conference on Pattern Recognition.

[135]  T. Metin Sezgin,et al.  Sketch interpretation using multiscale stochastic models of temporal patterns , 2006 .

[136]  Kenneth D. Forbus,et al.  Qualitative Spatial Interpretation of Course-of-Action Diagrams , 2000, AAAI/IAAI.

[137]  Randall Davis,et al.  Interactive learning of structural shape descriptions from automatically generated near-miss examples , 2006, IUI '06.

[138]  Hod Lipson,et al.  Correlation-based reconstruction of a 3D object from a single freehand sketch , 2007, SIGGRAPH Courses.

[139]  Frank M. Shipman,et al.  Integrating different perspectives on design rationale: Supporting the emergence of design rationale from design communication , 1997, Artificial Intelligence for Engineering Design, Analysis and Manufacturing.

[140]  George Stiny,et al.  Shape Grammars and the Generative Specification of Painting and Sculpture , 1971, IFIP Congress.

[141]  Elie Bienenstock,et al.  Compositionality, MDL Priors, and Object Recognition , 1996, NIPS.

[142]  Alberto Del Bimbo,et al.  Visual Image Retrieval by Elastic Matching of User Sketches , 1997, IEEE Trans. Pattern Anal. Mach. Intell..

[143]  James V. Mahoney,et al.  Three main concerns in sketch recognition and an approach to addressing them , 2002 .

[144]  Krzysztof Z. Gajos,et al.  Rascal - A Resource Manager for Multi Agent Systems in Smart Spaces , 2001, CEEMAS.

[145]  Andrew S. Forsberg,et al.  The music notepad , 1998, UIST '98.

[146]  Joaquim A. Jorge,et al.  JavaSketchIt: Issues in Sketching the Look of User Interfaces , 2002 .

[147]  Eric Saund,et al.  Perceptual organization in an interactive sketch editing application , 1995, Proceedings of IEEE International Conference on Computer Vision.

[148]  James Arvo,et al.  A Synthesis of Recognition and Morphing , 2000 .

[149]  Kim L. Boyer,et al.  Quantitative Measures of Change Based on Feature Organization: Eigenvalues and Eigenvectors , 1998, Comput. Vis. Image Underst..

[150]  Martin Odersky,et al.  Building visual language parsers , 1991, CHI '91.

[151]  Eric Saund,et al.  Finding Perceptually Closed Paths in Sketches and Drawings , 2003, IEEE Trans. Pattern Anal. Mach. Intell..

[152]  Stuart Russell,et al.  Statistical Visual Language Models for Ink Parsing , 2002 .

[153]  Max J. Egenhofer,et al.  Query Processing in Spatial-Query-by-Sketch , 1997, J. Vis. Lang. Comput..

[154]  Mark W. Newman,et al.  DENIM: An Informal Web Site Design Tool Inspired by Observations of Practice , 2003, Hum. Comput. Interact..

[155]  Peter J. Stuckey,et al.  The Cassowary linear arithmetic constraint solving algorithm , 2001, TCHI.

[156]  Nick Hammond,et al.  Graphical Argumentation and Design Cognition , 1997, Hum. Comput. Interact..

[157]  Jacob Eisenstein,et al.  A Salience-Based Approach to Gesture-Speech Alignment , 2004, HLT-NAACL.

[158]  Avrim Blum,et al.  The Bottleneck , 2021, Monopsony Capitalism.

[159]  Jake K. Aggarwal,et al.  CAD-based vision: object recognition in cluttered range images using recognition strategies , 1993 .

[160]  E Goldmeier,et al.  Similarity in visually perceived forms. , 1972, Psychological issues.

[161]  Thomas F. Stahovich,et al.  Generating Multiple New Designs from a Sketch , 1996, Artif. Intell..

[162]  L. Ball,et al.  Structure in idea sketching behaviour , 1998 .

[163]  Sean Jy-Shyang Chen,et al.  An interactive system for recognizing hand drawn UML diagrams , 2000, CASCON.

[164]  Sharon L. Oviatt,et al.  Unification-based Multimodal Integration , 1997, ACL.

[165]  Jacob Eisenstein,et al.  Gesture Features for Sentence Segmentation , 2005 .

[166]  Mark W. Newman,et al.  DENIM: an informal tool for early stage web site design , 2001, CHI Extended Abstracts.

[167]  David Pugh,et al.  Designing solid objects using interactive sketch interpretation , 1992, I3D '92.

[168]  Bo Yu,et al.  A domain-independent system for sketch recognition , 2003, GRAPHITE '03.

[169]  Jacob Eisenstein,et al.  Natural gesture in descriptive monologues , 2006, SIGGRAPH Courses.

[170]  Jacob Eisenstein,et al.  Multimodal Alignment by Optimization , 2003 .

[171]  Robert J. K. Jacob,et al.  A software model and specification language for non-WIMP user interfaces , 1999, TCHI.

[172]  Tom Michael Mitchell.  Version spaces: an approach to concept learning. , 1979 .

[173]  Christine Alvarado,et al.  Multi-Domain Sketch Recognition , 2002 .

[174]  John S. Gero,et al.  The structure of concurrent cognitive actions: a case study on novice and expert designers , 2002 .

[175]  M. Sakauchi,et al.  A drawing recognition system with rule acquisition ability , 1995, Proceedings of 3rd International Conference on Document Analysis and Recognition.

[176]  Alessandro Micarelli,et al.  Content Based Image Retrieval for Unsegmented Images , 2003, AI*IA.

[177]  Arne Jönsson,et al.  Wizard of Oz studies: why and how , 1993, IUI '93.

[178]  Frank Van Reeth,et al.  A Multi-Level Sketching Tool for "Pencil-and-Paper" Animation , 2002 .

[179]  Randall Davis,et al.  Automatically Transforming Symbolic Shape Descriptions for Use in Sketch Recognition , 2004, AAAI.

[180]  David Salesin,et al.  Fast multiresolution image querying , 1995, SIGGRAPH.

[181]  Oliver Bimber,et al.  A multi-layered architecture for sketch-based interaction within virtual environments , 2000, Comput. Graph..

[182]  Richard A. Bolt,et al.  “Put-that-there”: Voice and gesture at the graphics interface , 1980, SIGGRAPH '80.

[183]  Richard J. Anderson,et al.  Experiences with a tablet PC based lecture presentation system in computer science courses , 2004, SIGCSE '04.

[184]  Lippold Haken,et al.  Using Diagram Generation Software to Improve Diagram Recognition: A Case Study of Music Notation , 1999, IEEE Trans. Pattern Anal. Mach. Intell..

[185]  Tracy Anne Hammond,et al.  LADDER: a language to describe drawing, display, and editing in sketch recognition , 2003, IJCAI.

[186]  James F. Allen,et al.  Towards Conversational Human-Computer Interaction , 2000 .

[187]  Kenneth D. Forbus,et al.  Open-Domain Sketch Understanding: The nuSketch Approach , 2004, AAAI Technical Report.