Reverse Engineering Mobile Application User Interfaces with REMAUI (T)

When developing the user interface code of a mobile application, in practice a big gap exists between the digital conceptual drawings of graphic artists and working user interface code. Currently, programmers bridge this gap manually, by reimplementing the conceptual drawings in code, which is cumbersome and expensive. To bridge this gap, we introduce the first technique to automatically Reverse Engineer Mobile Application User Interfaces (REMAUI). On a given input bitmap REMAUI identifies user interface elements such as images, texts, containers, and lists, via computer vision and optical character recognition (OCR) techniques. In our experiments on 488 screenshots of over 100 popular third-party Android and iOS applications, REMAUI-generated user interfaces were similar to the originals, both pixel-by-pixel and in terms of their runtime user interface hierarchies. REMAUI's average overall runtime on a standard desktop computer was 9 seconds.

[1]  Nuno Jardim Nunes,et al.  Practitioner Tools and Workstyles for User-Interface Design , 2007, IEEE Software.

[2]  Gary R. Bradski,et al.  Learning OpenCV - computer vision with the OpenCV library: software that sees , 2008 .

[3]  William G. Griswold,et al.  Dynamically discovering likely program invariants to support program evolution , 1999, Proceedings of the 1999 International Conference on Software Engineering (IEEE Cat. No.99CB37002).

[4]  Morgan Dixon,et al.  Content and hierarchy in pixel-based methods for reverse engineering interface structure , 2011, CHI.

[5]  Greg Nudelman.  Android Design Patterns: Interaction Design Solutions for Developers , 2013 .

[6]  Anil K. Jain,et al.  Feature extraction methods for character recognition-A survey , 1996, Pattern Recognit..

[7]  Wolfgang Slany,et al.  User Interface Design for a Mobile Multimedia Application: An Iterative Approach , 2008, First International Conference on Advances in Computer-Human Interaction.

[8]  Volker Gruhn,et al.  Modeling Web-based dialog flows for automatic dialog control , 2004, Proceedings. 19th International Conference on Automated Software Engineering, 2004..

[9]  Apostolos Antonacopoulos,et al.  ICDAR 2009 Page Segmentation Competition , 2003, 2009 10th International Conference on Document Analysis and Recognition.

[10]  Alex Zelinsky,et al.  Learning OpenCV---Computer Vision with the OpenCV Library (Bradski, G.R. et al.; 2008)[On the Shelf] , 2009, IEEE Robotics & Automation Magazine.

[11]  Masumi Nakamura,et al.  Learning Android: Develop Mobile Apps Using Java and Eclipse , 2014 .

[12]  Jon Almazán,et al.  ICDAR 2013 Robust Reading Competition , 2013, 2013 12th International Conference on Document Analysis and Recognition.

[13]  Yuriy Brun,et al.  Finding latent code errors via machine learning over program executions , 2004, Proceedings. 26th International Conference on Software Engineering.

[14]  Raymond W. Smith.  Hybrid Page Layout Analysis via Tab-Stop Detection , 2009, 2009 10th International Conference on Document Analysis and Recognition.

[15]  Sargur N. Srihari,et al.  On-Line and Off-Line Handwriting Recognition: A Comprehensive Survey , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[16]  John C. Grundy,et al.  Model-Driven Development of Mobile Personal Health Care Applications , 2008, 2008 23rd IEEE/ACM International Conference on Automated Software Engineering.

[17]  Vladimir I. Levenshtein,et al.  Binary codes capable of correcting deletions, insertions, and reversals , 1965 .

[18]  Brad A. Myers,et al.  Graphical User Interface Programming , 2004 .

[19]  Alireza Sahami Shirazi,et al.  Insights into layout patterns of mobile user interfaces by an automatic analysis of android apps , 2013, EICS '13.

[20]  Yannis Smaragdakis,et al.  Dynamically discovering likely interface invariants , 2006, ICSE '06.

[21]  Tom Yeh,et al.  Associating the visual representation of user interfaces with their internal structures and metadata , 2011, UIST.

[22]  L. Fortunati,et al.  The advanced use of mobile phones in five European countries. , 2014, The British journal of sociology.

[23]  Mark W. Newman,et al.  Sitemaps, storyboards, and specifications: a sketch of Web site design practice , 2000, DIS '00.

[24]  Brad A. Myers,et al.  Past, Present and Future of User Interface Software Tools , 2000, TCHI.

[25]  Albrecht Schmidt,et al.  Mobidev: a tool for creating apps on mobile phones , 2011, Mobile HCI.

[26]  Yin Yin Wong.  Rough and ready prototypes: lessons from graphic design , 1992, CHI '92.

[27]  James A. Landay,et al.  Sketching Interfaces: Toward More Human Interface Design , 2001, Computer.

[28]  Richard Szeliski,et al.  Computer Vision - Algorithms and Applications , 2011, Texts in Computer Science.

[29]  Edin Arnautovic,et al.  Using communicative acts in interaction design specifications for automated synthesis of user interfaces , 2006, 21st IEEE/ACM International Conference on Automated Software Engineering (ASE'06).

[30]  Steven P. Reiss.  Seeking the user interface , 2014, ASE.

[31]  Subramanian Ramanathan,et al.  WADE: simplified GUI add-on development for third-party software , 2014, CHI.

[32]  Jean Vanderdonckt,et al.  Multi-fidelity Prototyping of User Interfaces , 2007, INTERACT.

[33]  Basilios Gatos,et al.  ICDAR2005 page segmentation competition , 2007, Eighth International Conference on Document Analysis and Recognition (ICDAR'05).

[34]  Oscar Pastor,et al.  From early requirements to user interface prototyping: a methodological approach , 2002, Proceedings 17th IEEE International Conference on Automated Software Engineering,.

[35]  James H. Cross,et al.  Reverse engineering and design recovery: a taxonomy , 1990, IEEE Software.

[36]  Frank Maurer,et al.  User-Centered Design and Agile Methods: A Systematic Review , 2011, 2011 AGILE Conference.

[37]  John F. Canny,et al.  A Computational Approach to Edge Detection , 1986, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[38]  Christof Lutteroth,et al.  The Auckland layout editor: an improved GUI layout specification process , 2012, CHINZ '12.

[39]  James A. Landay,et al.  Interactive sketching for the early stages of user interface design , 1995, CHI '95.

[40]  Paul Johns,et al.  Working Overtime: Patterns of Smartphone and PC Usage in the Day of an Information Worker , 2009, Pervasive.

[41]  Joaquim A. Jorge,et al.  JavaSketchIt: Issues in Sketching the Look of User Interfaces , 2002 .

[42]  Luís Carriço,et al.  A mixed-fidelity prototyping tool for mobile devices , 2008, AVI '08.

[43]  Nikolai Tillmann,et al.  DySy: dynamic symbolic execution for invariant inference , 2008, ICSE.

[44]  Morgan Dixon,et al.  Prefab: implementing advanced behaviors using pixel-based reverse engineering of interface structure , 2010, CHI.

[45]  Nikolai Tillmann,et al.  GROPG: A graphical on-phone debugger , 2013, 2013 35th International Conference on Software Engineering (ICSE).

[46]  John Grundy,et al.  Bootstrapping Mobile App Development , 2015, 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering.

[47]  Brad A. Myers,et al.  Challenges of HCI design and implementation , 1994, INTR.

[48]  Shumin Zhai,et al.  Smart phone use by non-mobile business users , 2011, Mobile HCI.

[49]  Ying Zou,et al.  An Exploratory Study on the Relation between User Interface Complexity and the Perceived Quality , 2014, ICWE.

[50]  Tommi Mikkonen,et al.  Designing User Experience for Mobile Apps: Long-Term Product Owner Perspective , 2013, 2013 20th Asia-Pacific Software Engineering Conference (APSEC).

[51]  Stéphane Chatty,et al.  Revisiting visual interface programming: creating GUI tools for designers and programmers , 2004, UIST '04.

[52]  R. Smith,et al.  An Overview of the Tesseract OCR Engine , 2007, Ninth International Conference on Document Analysis and Recognition (ICDAR 2007).

[53]  Christof Lutteroth,et al.  Evaluating Direct Manipulation Operations for Constraint-Based Layout , 2013, INTERACT.