SKETRACK: Stroke-Based Recognition of Online Hand-Drawn Sketches of Arrow-Connected Diagrams and Digital Logic Circuit Diagrams

Digitalization of handwritten documents has created a greater need for accurate online recognition of hand-drawn sketches. However, the online recognition of hand-drawn diagrams is an enduring challenge in human-computer interaction due to the complexity in extracting and recognizing the visual objects reliably from a continuous stroke stream. This paper focuses on the design and development of a new, efficient stroke-based online hand-drawn sketch recognition scheme named SKETRACK for hand-drawn arrow diagrams and digital logic circuit diagrams. The fundamental parts of this model are text separation, symbol segmentation, feature extraction, classification, and structural analysis. The proposed scheme utilizes the concepts of normalization and segmentation to isolate the text from the sketches. Then, the features are extracted to model different structural variations of the strokes that are categorized into the arrows/lines and the symbols for effective processing. The strokes are clustered using the spectral clustering algorithm based on p-distance and Euclidean distance to compute the similarity between the features and minimize the feature dimensionality by grouping similar features. Then, the symbol recognition is performed using modified support vector machine (MSVM) classifier in which a hybrid kernel function with a lion optimized tuning parameter of SVM is utilized. Structural analysis is performed with lion-based task optimization for recognizing the symbol candidates to form the final diagram representations. This proposed recognition model is suitable for simpler structures such as flowcharts, finite automata, and the logic circuit diagrams. Through the experiments, the performance of the proposed SKETRACK scheme is evaluated on three domains of databases and the results are compared with the state-of-the-art methods to validate its superior efficiency.

[1]  Marc Alexa,et al.  Sketch-Based Image Retrieval: Benchmark and Bag-of-Features Descriptors , 2011, IEEE Transactions on Visualization and Computer Graphics.

[2]  Lei Li Fast Sketch Segmentation and Labeling with Deep Learning , 2018 .

[3]  Ioannis A. Kakadiaris,et al.  Three-Dimensional Face Recognition in the Presence of Facial Expressions: An Annotated Deformable Model Approach , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[4]  Marc Alexa,et al.  How do humans sketch objects? , 2012, ACM Trans. Graph..

[5]  Tinne Tuytelaars,et al.  Sketch classification and classification-driven analysis using Fisher vectors , 2014, ACM Trans. Graph..

[6]  Wioleta Szwoch,et al.  Recognition of Hand Drawn Flowcharts , 2012, International Conference on Image Processing and Communications Challenges.

[7]  Chao Tian,et al.  Physical sketching: Reconstruction and analysis of 3D objects from freehand sketches , 2009, Comput. Aided Des..

[8]  Liqing Zhang,et al.  Free Hand-Drawn Sketch Segmentation , 2012, ECCV.

[9]  Xiaochun Cao,et al.  SYM-FISH: A Symmetry-Aware Flip Invariant Sketch Histogram Shape Descriptor , 2013, 2013 IEEE International Conference on Computer Vision.

[10]  Mustafa Sert,et al.  Feature-level fusion of deep convolutional neural networks for sketch recognition on smartphones , 2017, 2017 IEEE International Conference on Consumer Electronics (ICCE).

[11]  Xuejin Chen,et al.  Sketchpointnet: A Compact Network for Robust Sketch Recognition , 2018, 2018 25th IEEE International Conference on Image Processing (ICIP).

[12]  Liqing Zhang,et al.  Query-adaptive shape topic mining for hand-drawn sketch recognition , 2012, ACM Multimedia.

[13]  Tracy Anne Hammond,et al.  A Sketch Recognition System for Recognizing Free-Hand Course of Action Diagrams , 2010, IAAI.

[14]  Dhanasekaran Raghavan,et al.  Brain tumor severity analysis using modified multi‐texton histogram and hybrid kernel SVM , 2014, Int. J. Imaging Syst. Technol..

[15]  Madhuri S. Joshi,et al.  Hand-drawn Digital Logic Circuit Component Recognition using SVM , 2016 .

[16]  Yong Jae Lee,et al.  ShadowDraw: real-time user guidance for freehand drawing , 2011, SIGGRAPH 2011.

[17]  Marcus Liwicki,et al.  IAMonDo-database: an online handwritten document database with non-uniform contents , 2010, DAS '10.

[18]  Stéphane Dupont,et al.  DeepSketch: Deep convolutional neural networks for sketch recognition and similarity search , 2015, 2015 13th International Workshop on Content-Based Multimedia Indexing (CBMI).

[19]  Manolya Kavakli,et al.  MATRACK: block sparse Bayesian learning for a sketch recognition approach , 2017, Multimedia Tools and Applications.

[20]  James Hays,et al.  The sketchy database , 2016, ACM Trans. Graph..

[21]  Feng Liu,et al.  Sketch Me That Shoe , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[22]  Volker Gruhn,et al.  Evaluation of a traceability approach for informal freehand sketches , 2018, Automated Software Engineering.

[23]  Bo Zhao,et al.  Sketch Recognition with Deep Visual-Sequential Fusion Model , 2017, ACM Multimedia.

[24]  Václav Hlavác,et al.  Recognition System for On-Line Sketched Diagrams , 2014, 2014 14th International Conference on Frontiers in Handwriting Recognition.

[25]  Liqing Zhang,et al.  Edgel index for large-scale sketch-based image search , 2011, CVPR 2011.

[26]  Tao Xiang,et al.  Sketch-a-Net: A Deep Neural Network that Beats Humans , 2017, International Journal of Computer Vision.

[27]  Omar Seddati,et al.  DeepSketch 2: Deep convolutional neural networks for partial sketch recognition , 2016, 2016 14th International Workshop on Content-Based Multimedia Indexing (CBMI).

[28]  David Gur,et al.  Automated freehand sketch segmentation using radial basis functions , 2009, Comput. Aided Des..

[29]  Václav Hlavác,et al.  Online recognition of sketched arrow-connected diagrams , 2016, International Journal on Document Analysis and Recognition (IJDAR).

[30]  Shaogang Gong,et al.  Free-Hand Sketch Synthesis with Deformable Stroke Models , 2016, International Journal of Computer Vision.

[31]  Hongxun Yao,et al.  Discovering discriminative patches for free-hand sketch analysis , 2017, Multimedia Systems.

[32]  Ravi Kiran Sarvadevabhatla,et al.  Enabling My Robot To Play Pictionary: Recurrent Neural Networks For Sketch Recognition , 2016, ACM Multimedia.

[33]  Hongyang Chao,et al.  Building Effective Representations for Sketch Recognition , 2015, AAAI.

[34]  Qi Zou,et al.  A Hybrid convolutional neural network for sketch recognition , 2020, Pattern Recognit. Lett..

[35]  Stéphane Dupont,et al.  DeepSketch 3 , 2017, Multimedia Tools and Applications.

[36]  Fariborz Jolai,et al.  Lion Optimization Algorithm (LOA): A nature-inspired metaheuristic algorithm , 2016, J. Comput. Des. Eng..

[37]  Yan Zhang,et al.  On the Euclidean distance of images , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[39]  Ke Wang,et al.  A fast incremental map segmentation algorithm based on spectral clustering and quadtree , 2018 .

[40]  Yong Jae Lee,et al.  ShadowDraw: real-time user guidance for freehand drawing , 2011, ACM Trans. Graph..

[41]  Mustafa Sert,et al.  Sketch recognition using transfer learning , 2018, Multimedia Tools and Applications.

[42]  Heiko Schuldt,et al.  Cineast: A Multi-feature Sketch-Based Video Retrieval Engine , 2014, 2014 IEEE International Symposium on Multimedia.

[43]  Soonhung Han,et al.  A procedural method to exchange editable 3D data from a free-hand 2D sketch modeling system into 3D mechanical CAD systems , 2012, Comput. Aided Des..

[44]  Beryl Plimmer,et al.  Supervised machine learning for grouping sketch diagram strokes , 2013, SBIM '13.

[45]  Shaogang Gong,et al.  Free-hand sketch recognition by multi-kernel feature learning , 2015, Comput. Vis. Image Underst..

[46]  Geoffrey E. Hinton,et al.  Speech recognition with deep recurrent neural networks , 2013, 2013 IEEE International Conference on Acoustics, Speech and Signal Processing.

[47]  Shaogang Gong,et al.  Sketch Recognition by Ensemble Matching of Structured Features , 2013, BMVC.

[48]  Rynson W. H. Lau,et al.  Data-driven segmentation and labeling of freehand sketches , 2014, ACM Trans. Graph..