Sketch recognition by fusion of temporal and image-based features

The increasing availability of pen-based hardware has recently resulted in a parallel growth in sketch-based user interfaces. Sketch-based user interfaces aim to combine the expressive power of free-hand sketching with the processing power of computers. Most sketch-based systems require intelligent ink processing capabilities, which makes the development of robust sketch recognition algorithms a primary concern in the field. So far, the research in sketch recognition has produced various independent approaches to recognition, each of which uses a particular kind of information (e.g., geometric and spatial constraints, image-based features, temporal stroke-ordering patterns). These methods were designed in isolation as stand-alone algorithms, and there has been little work treating various recognition methods as alternative sources of information that can be combined to increase sketch recognition accuracy. In this paper, we focus on two such methods and fuse an image-based method with a time-based method in an attempt to combine the knowledge of how objects look (image data) with the knowledge of how they are drawn (temporal data). In the course of combining spatial and temporal information, we also introduce a mathematically well founded fusion method for combining recognizers. Our combination method can be used for isolated sketch recognition as well as full diagram recognition. Our evaluation with two databases shows that fusing image-based and temporal features yields higher recognition rates. These results are the first to confirm the complementary nature of image-based and temporal recognition methods for full sketch recognition, which has long been suggested, but never supported by data.

[1]  Gregory Dudek,et al.  Sketch Interpretation and Refinement Using Statistical Models , 2004, Rendering Techniques.

[2]  Michael Collins,et al.  Discriminative Training Methods for Hidden Markov Models: Theory and Experiments with Perceptron Algorithms , 2002, EMNLP.

[3]  Adam Krzyżak,et al.  Methods of combining multiple classifiers and their applications to handwriting recognition , 1992, IEEE Trans. Syst. Man Cybern..

[4]  M. Cheriet,et al.  Online recognition of sketched electrical diagrams , 2001, Proceedings of Sixth International Conference on Document Analysis and Recognition.

[5]  Levent Burak Kara,et al.  Combining geometry and domain knowledge to interpret hand-drawn diagrams , 2005, Comput. Graph..

[6]  Marjorie Skubic,et al.  Hidden Markov Model Symbol Recognition for Sketch-Based Interfaces , 2004, AAAI Technical Report.

[7]  Randall Davis,et al.  HMM-based efficient sketch recognition , 2005, IUI.

[8]  Mario Vento,et al.  Combining statistical and structural approaches for handwritten character description , 1999, Image Vis. Comput..

[9]  Paul A. Viola,et al.  Spatial recognition and grouping of text and graphics , 2004, SBM'04.

[10]  Michael Oltmans.  Envisioning sketch recognition: a local feature based approach to recognizing informal sketches , 2007 .

[11]  Chih-Jen Lin,et al.  LIBSVM: A library for support vector machines , 2011, TIST.

[12]  Levent Burak Kara,et al.  An Image-Based Trainable Symbol Recognizer for Sketch-Based Interfaces , 2004, AAAI Technical Report.

[13]  Hsuan-Tien Lin,et al.  A note on Platt’s probabilistic outputs for support vector machines , 2007, Machine Learning.

[14]  Jitendra Malik,et al.  Shape matching and object recognition using shape contexts , 2010, 2010 3rd International Conference on Computer Science and Information Technology.

[15]  Éric Anquetil,et al.  Eager interpretation of on-line hand-drawn structured documents: The DALI methodology , 2009, Pattern Recognit..

[16]  Vladimir Vapnik,et al.  Statistical learning theory , 1998 .

[17]  Michiel van de Panne,et al.  Constellation models for sketch recognition , 2006, SBM'06.

[18]  Erik G. Learned-Miller,et al.  Data driven image models through continuous joint alignment , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[19]  Marcus Liwicki,et al.  Combining diverse systems for handwritten text line recognition , 2009, Machine Vision and Applications.

[20]  Louis Vuurpijl,et al.  Iconic and multi-stroke gesture recognition , 2009, Pattern Recognit..

[21]  K. Ishigaki,et al.  Hybrid pen-input character recognition system based on integration of online-offline recognition , 1999, Proceedings of the Fifth International Conference on Document Analysis and Recognition. ICDAR '99 (Cat. No.PR00318).

[22]  A. Richard Newton,et al.  Sketched symbol recognition using Zernike moments , 2004, Proceedings of the 17th International Conference on Pattern Recognition, 2004. ICPR 2004..

[23]  Yuan Qi,et al.  Diagram structure recognition by Bayesian conditional random fields , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[24]  Stuart Russell,et al.  Statistical Visual Language Models for Ink Parsing , 2002 .

[25]  Randall Davis,et al.  Sketch recognition in interspersed drawings using time-based graphical models , 2008, Comput. Graph..

[26]  Noël Richard,et al.  Collaboration Between Statistical and Structural Approaches for Old Handwritten Characters Recognition , 2005, GbRPR.

[27]  T. F. Sezgin.  Genetic and HMM based ap-proaches to freehand sketch recognition , 2003 .

[28]  Christian Viard-Gaudin,et al.  On-line hand-drawn electric circuit diagram recognition using 2D dynamic programming , 2009, Pattern Recognit..

[29]  Jiri Matas,et al.  On Combining Classifiers , 1998, IEEE Trans. Pattern Anal. Mach. Intell..

[30]  Marcus Liwicki,et al.  HMM-Based On-Line Recognition of Handwritten Whiteboard Notes , 2006 .

[31]  Martin Szummer,et al.  A Graphical Model for Simultaneous Partitioning and Labeling , 2005, AISTATS.

[32]  Dean Rubine,et al.  Specifying gestures by example , 1991, SIGGRAPH.

[33]  Marcus Liwicki,et al.  Combining diverse on-line and off-line systems for handwritten text line recognition , 2009, Pattern Recognit..

[34]  Fuad Rahman,et al.  Multiple classifier decision combination strategies for character recognition: A review , 2003, Document Analysis and Recognition.

[35]  Randall Davis,et al.  LADDER, a sketching language for user interface developers , 2005, Comput. Graph..

[36]  Bernhard Schölkopf,et al.  Estimating the Support of a High-Dimensional Distribution , 2001, Neural Computation.

[37]  Christine Alvarado,et al.  SketchREAD: a multi-domain sketch recognition engine , 2004, UIST '04.

[38]  P ? ? ? ? ? ? ? % ? ? ? ? , 1991 .

[39]  Lalit R. Bahl,et al.  Maximum mutual information estimation of hidden Markov model parameters for speech recognition , 1986, ICASSP '86. IEEE International Conference on Acoustics, Speech, and Signal Processing.

[40]  Randall Davis,et al.  Recognizing interspersed sketches quickly , 2009, Graphics Interface.

[41]  Alexander Hall,et al.  A combinatorial approach to multi-domain sketch recognition , 2007, SBIM '07.

[42]  Biing-Hwang Juang,et al.  Fundamentals of speech recognition , 1993, Prentice Hall signal processing series.

[43]  Cheng Wang,et al.  Adaptive binary tree for fast SVM multiclass classification , 2009, Neurocomputing.

[44]  Bart Lamiroy,et al.  An Incremental On-line Parsing Algorithm for Recognizing Sketching Diagrams , 2007, Ninth International Conference on Document Analysis and Recognition (ICDAR 2007).

[45]  Wei Jiang,et al.  HMM-based on-line multi-stroke sketch recognition , 2005, 2005 International Conference on Machine Learning and Cybernetics.

[46]  Lawrence R. Rabiner,et al.  A tutorial on hidden Markov models and selected applications in speech recognition , 1989, Proc. IEEE.

[47]  Masaki Nakagawa,et al.  A Compact On-line and Off-line Combined Recognizer , 2006 .

[48]  Ehud Rivlin,et al.  An integration of online and pseudo-online information for cursive word recognition , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[49]  Nicu Sebe,et al.  Multimodal Human Computer Interaction: A Survey , 2005, ICCV-HCI.

[50]  C. Alvarado,et al.  Properties of Real-World Digital Logic Diagrams , 2007, First International Workshop on Pen-Based Learning Technologies (PLT 2007).

[51]  Marcus Liwicki,et al.  A novel approach to on-line handwriting recognition based on bidirectional long short-term memory networks , 2007 .