Multi-Domain Recognition of Hand-Drawn Diagrams Using Hierarchical Parsing

This paper presents an approach for the recognition of multi-domain hand-drawn diagrams, which exploits Sketch Grammars (SkGs) to model the symbols’ shape and the abstract syntax of diagrammatic notations. The recognition systems automatically generated from SkGs process the input sketches according to the following phases: the user’ strokes are first segmented and interpreted as primitive shapes, then by exploiting the domain context they are clustered into symbols of the domain and, finally, an interpretation of whole diagram is given. The main contribution of this paper is an efficient model of parsing suitable for both interactive and non-interactive sketch-based interfaces, configurable to different domains, and able to exploit contextual information for improving recognition accuracy and solving interpretation ambiguities. The proposed approach was evaluated in the domain of UML class diagrams obtaining good results in terms of recognition accuracy and usability.

[1]  Qi Zou,et al.  A Hybrid convolutional neural network for sketch recognition , 2020, Pattern Recognit. Lett..

[2]  Randall Davis,et al.  Magic Paper: Sketch-Understanding Research , 2007, Computer.

[3]  Michele Risi,et al.  Sketched symbol recognition using Latent-Dynamic Conditional Random Fields and distance-based clustering , 2014, Pattern Recognit..

[4]  Randall Davis,et al.  LADDER, a sketching language for user interface developers , 2005, Comput. Graph..

[5]  Takeo Igarashi,et al.  Guest Editors' Introduction: Sketch-Based Interaction , 2007, IEEE Computer Graphics and Applications.

[6]  Paul Asente,et al.  Advanced drawing beautification with ShipShape , 2016, Comput. Graph..

[7]  Randall Davis,et al.  Sketch Interpretation Using Multiscale Models of Temporal Patterns , 2007, IEEE Computer Graphics and Applications.

[8]  Joan-Andreu Sánchez,et al.  Estimation of stochastic context-free grammars and their use as language models , 2005, Comput. Speech Lang..

[9]  Eric Saund,et al.  Sloppy selection: Providing an accurate interpretation of imprecise selection gestures , 2005, Comput. Graph..

[10]  Michael R. Lyu,et al.  Extraction of line segments and circular arcs from freehand strokes based on segmental homogeneity features , 2006, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[11]  Michele Risi,et al.  Building syntax-aware editors for visual languages , 2005, J. Vis. Lang. Comput..

[12]  Tinne Tuytelaars,et al.  Example-Based Sketch Segmentation and Labeling Using CRFs , 2016, ACM Trans. Graph..

[13]  Gregor Snelting,et al.  How to build LR parsers which accept incomplete input , 1990, SIGP.

[14]  Levent Burak Kara,et al.  An image-based, trainable symbol recognizer for hand-drawn sketches , 2005, Comput. Graph..

[15]  Thomas F. Stahovich,et al.  Generating Multiple New Designs from a Sketch , 1996, Artif. Intell..

[16]  Jianyong Sun,et al.  Adaptive Online Multi-stroke Sketch Recognition Based on Hidden Markov Model , 2005, ICMLC.

[17]  Lei Li,et al.  Fast Sketch Segmentation and Labeling With Deep Learning , 2018, IEEE Computer Graphics and Applications.

[18]  Aurélie Lemaitre,et al.  Interactive interpretation of structured documents: Application to the recognition of handwritten architectural plans , 2015, Pattern Recognit..

[19]  Joan-Andreu Sánchez,et al.  Recognition of on-line handwritten mathematical expressions using 2D stochastic context-free grammars and hidden Markov models , 2014, Pattern Recognit. Lett..

[20]  Václav Hlavác,et al.  Online recognition of sketched arrow-connected diagrams , 2016, International Journal on Document Analysis and Recognition (IJDAR).

[21]  Éric Anquetil,et al.  Eager interpretation of on-line hand-drawn structured documents: The DALI methodology , 2009, Pattern Recognit..

[22]  Dong-Yul Ra,et al.  A New one Pass Algorithm for Estimating Stochastic Context-Free Grammars , 1999, Inf. Process. Lett..

[23]  RubineDean.  Specifying gestures by example , 1991 .

[24]  Marti A. Hearst.  Sketching intelligent systems , 1998 .

[25]  Gennaro Costagliola,et al.  A framework for modeling and implementing visual notations with applications to software engineering , 2004, TSEM.

[26]  Tinne Tuytelaars,et al.  Sketch classification and classification-driven analysis using Fisher vectors , 2014, ACM Trans. Graph..

[27]  Erin McTigue,et al.  Mechanix: A natural sketch interface tool for teaching truss analysis and free-body diagrams , 2014, Artificial Intelligence for Engineering Design, Analysis and Manufacturing.