Doodle2App: native app code by freehand UI sketching

User interface development typically starts with freehand sketching, with pen on paper, which creates a big gap in the software development process. Recent advances in deep neural networks that have been trained on large sketch stroke sequence collections have enabled online sketch detection that supports many sketch element classes at high classification accuracy. This paper leverages the recent Google Quick, Draw! dataset of 50M sketch stroke sequences to pre-train a recurrent neural network and retrains it with sketch stroke sequences we collected via Amazon Mechanical Turk. The resulting Doodle2App website offers a paper substitute, i.e., a drawing interface with interactive UI preview and can convert sketches to a compilable single-page Android application. On 712 sketch samples Doodle2App achieved higher accuracy than the state-of-the-art tool Teleport. A video demo is at https://youtu.be/P4sb0pKTNEY

[1]  Christopher D. Hundhausen,et al.  How is User Interface Prototyping Really Done in Practice? A Survey of User Interface Designers , 2010, 2010 IEEE Symposium on Visual Languages and Human-Centric Computing.

[2]  Joaquim A. Jorge,et al.  JavaSketchIt: Issues in Sketching the Look of User Interfaces , 2002 .

[3]  Gavriel Salomon,et al.  T RANSFER OF LEARNING , 1992 .

[4]  Luís Carriço,et al.  A mixed-fidelity prototyping tool for mobile devices , 2008, AVI '08.

[5]  Pietro Perona,et al.  Learning Generative Visual Models from Few Training Examples: An Incremental Bayesian Approach Tested on 101 Object Categories , 2004, 2004 Conference on Computer Vision and Pattern Recognition Workshop.

[6]  Yin Yin Wong.  Rough and ready prototypes: lessons from graphic design , 1992, CHI '92.

[7]  Jean Vanderdonckt,et al.  Multi-fidelity Prototyping of User Interfaces , 2007, INTERACT.

[8]  Emilio Soria Olivas,et al.  Handbook of Research on Machine Learning Applications and Trends : Algorithms , Methods , and Techniques , 2009 .

[9]  Mark W. Newman,et al.  Sitemaps, storyboards, and specifications: a sketch of Web site design practice , 2000, DIS '00.

[10]  James Hays,et al.  The sketchy database , 2016, ACM Trans. Graph..

[11]  Kuldip K. Paliwal,et al.  Bidirectional recurrent neural networks , 1997, IEEE Trans. Signal Process..

[12]  James A. Landay,et al.  Sketching Interfaces: Toward More Human Interface Design , 2001, Computer.

[13]  Jeffrey Nichols,et al.  Rico: A Mobile App Dataset for Building Data-Driven Design Applications , 2017, UIST.

[14]  Nuno Jardim Nunes,et al.  Practitioner Tools and Workstyles for User-Interface Design , 2007, IEEE Software.

[15]  James A. Landay,et al.  Interactive sketching for the early stages of user interface design , 1995, CHI '95.

[16]  Tuan Anh Nguyen,et al.  Reverse Engineering Mobile Application User Interfaces with REMAUI (T) , 2015, 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE).

[17]  Douglas Eck,et al.  A Neural Representation of Sketch Drawings , 2017, ICLR.

[18]  Zachary Chase Lipton.  A Critical Review of Recurrent Neural Networks for Sequence Learning , 2015, ArXiv.

[19]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[20]  Yuan Yu,et al.  TensorFlow: A system for large-scale machine learning , 2016, OSDI.

[21]  Albrecht Schmidt,et al.  Mobidev: a tool for creating apps on mobile phones , 2011, Mobile HCI.