Rewire: Interface Design Assistance from Examples

Interface designers often use screenshot images of example designs as building blocks for new designs. Since images are unstructured and hard to edit, designers typically reconstruct screenshots with vector graphics tools in order to reuse or edit parts of the design. Unfortunately, this reconstruction process is tedious and slow. We present Rewire, an interactive system that helps designers leverage example screenshots. Rewire automatically infers a vector representation of screenshots where each UI component is a separate object with editable shape and style properties. Based on this representation, the system provides three design assistance modes that help designers reuse or redraw components of the example design. The results from our quantitative and user evaluations demonstrate that Rewire can generate accurate vector representations of interface screenshots found in the wild and that design assistance enables users to reconstruct and edit example designs more efficiently compared to a baseline design tool.

[1]  Gierad Laput,et al.  Pixel-based methods for widget state and style in a runtime implementation of sliding widgets , 2014, CHI.

[2]  Luis Miguel Bergasa,et al.  Text location in complex images , 2012, Proceedings of the 21st International Conference on Pattern Recognition (ICPR2012).

[3]  Ronen I. Brafman,et al.  Designing with interactive example galleries , 2010, CHI.

[4]  Morgan Dixon,et al.  Content and hierarchy in pixel-based methods for reverse engineering interface structure , 2011, CHI.

[5]  Takeo Igarashi,et al.  Global beautification of layouts with interactive ambiguity resolution , 2014, UIST.

[6]  R. Smith,et al.  An Overview of the Tesseract OCR Engine , 2007, Ninth International Conference on Document Analysis and Recognition (ICDAR 2007).

[7]  Aaron Hertzmann,et al.  DesignScape: Design with Interactive Layout Suggestions , 2015, CHI.

[8]  Roberto Cipolla,et al.  SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[9]  Luis M. Bergasa,et al.  Location in Complex Images , 2012 .

[10]  Bill Buxton,et al.  Sketching User Experiences: Getting the Design Right and the Right Design , 2007 .

[11]  Tony Beltramelli,et al.  pix2code: Generating Code from a Graphical User Interface Screenshot , 2017, EICS.

[12]  Daniel L. Schwartz,et al.  Parallel prototyping leads to better design results, more divergence, and increased self-efficacy , 2010, TCHI.

[13]  Nikolaj Bjørner,et al.  Z3: An Efficient SMT Solver , 2008, TACAS.

[14]  Wencheng Wu,et al.  The CIEDE2000 color-difference formula: Implementation notes, supplementary test data, and mathematical observations , 2005 .

[15]  Aaron Hertzmann,et al.  Learning Layouts for Single-PageGraphic Designs , 2014, IEEE Transactions on Visualization and Computer Graphics.

[16]  Ranjitha Kumar,et al.  Bricolage: example-based retargeting for web design , 2011, CHI.

[17]  Ranjitha Kumar,et al.  ERICA: Interaction Mining Mobile Apps , 2016, UIST.

[18]  Brian P. Bailey,et al.  Getting inspired!: understanding how and why examples are used in creative design practice , 2009, CHI.

[19]  Rob Miller,et al.  Sikuli: using GUI screenshots for search and automation , 2009, UIST '09.

[20]  Huchuan Lu,et al.  Scene text detection via stroke width , 2012, Proceedings of the 21st International Conference on Pattern Recognition (ICPR2012).

[21]  Dana H. Ballard,et al.  Generalizing the Hough transform to detect arbitrary shapes , 1981, Pattern Recognit..

[22]  Morgan Dixon,et al.  Prefab: implementing advanced behaviors using pixel-based reverse engineering of interface structure , 2010, CHI.

[23]  Lattre de Tassigny Boundary Extraction in Natural Images Using Ultrametric Contour Maps , 2006 .

[24]  Ranjitha Kumar,et al.  Webzeitgeist: design mining the web , 2013, CHI.

[25]  S. Holm A Simple Sequentially Rejective Multiple Test Procedure , 1979 .

[26]  Tuan Anh Nguyen,et al.  Reverse Engineering Mobile Application User Interfaces with REMAUI (T) , 2015, 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE).