What do all these Buttons do? Statically Mining Android User Interfaces at Scale

We introduce Frontmatter: a tool to automatically mine both user interface models and behavior of Android apps at a large scale with high precision. Given an app, Frontmatter statically extracts all declared screens, the user interface elements, their textual and graphical features, as well as Android APIs invoked by interacting with them. Executed on tens of thousands of real-world apps, Frontmatter opens the door for comprehensive mining of mobile user interfaces, jumpstarting empirical research at a large scale, addressing questions such as “How many travel apps require registration?”, “Which apps do not follow accessibility guidelines?”, “Does the user interface correspond to the description?”, and many more. Frontmatter and the mined dataset are available under an open-source license.

[1]  Eric Bodden,et al.  Context-, flow-, and field-sensitive data-flow analysis using synchronized Pushdown systems , 2019, Proc. ACM Program. Lang..

[2]  Alessandro Orso,et al.  GUIFetch: Supporting App Design and Development through GUI Search , 2018, 2018 IEEE/ACM 5th International Conference on Mobile Software Engineering and Systems (MOBILESoft).

[3]  Jeffrey Nichols,et al.  Rico: A Mobile App Dataset for Building Data-Driven Design Applications , 2017, UIST.

[4]  Alessandra Gorla,et al.  Detecting Behavior Anomalies in Graphical User Interfaces , 2017, 2017 IEEE/ACM 39th International Conference on Software Engineering Companion (ICSE-C).

[5]  Yan Wang,et al.  On the unsoundness of static analysis for Android GUIs , 2016, SOAP@PLDI.

[6]  Alireza Sadeghi,et al.  Reducing Combinatorics in GUI Testing of Android Applications , 2016, 2016 IEEE/ACM 38th International Conference on Software Engineering (ICSE).

[7]  Suman Nath,et al.  Brahmastra: Driving Apps to Test the Security of Third-Party Components , 2014, USENIX Security Symposium.

[8]  Ondrej Lhoták,et al.  Scaling Java Points-to Analysis Using SPARK , 2003, CC.

[9]  Tom Yeh,et al.  Collect, Decompile, Extract, Stats, and Diff: Mining Design Pattern Changes in Android Apps , 2015, MobileHCI.

[10]  Jin Wang,et al.  A Vision-Based Approach for Deep Web Form Extraction , 2017, MUE/FutureTech.

[11]  Thomas F. Liu,et al.  Learning Design Semantics for Mobile Apps , 2018, UIST.

[12]  Erwin Adi,et al.  Investigating Login Features in Smartphone Apps , 2018, UbiComp/ISWC Adjunct.

[13]  Wei Le,et al.  Generating Predicate Callback Summaries for the Android Framework , 2017, 2017 IEEE/ACM 4th International Conference on Mobile Software Engineering and Systems (MOBILESoft).

[14]  Ranjitha Kumar,et al.  Webzeitgeist: design mining the web , 2013, CHI.

[15]  Martin Gaedke,et al.  HCI Vision for Automated Analysis and Mining of Web User Interfaces , 2018, ICWE.

[16]  Lisa,et al.  Goal-driven Exploration for Android Applications , 2019 .

[17]  Denys Poshyvanyk,et al.  Machine Learning-Based Prototyping of Graphical User Interfaces for Mobile Apps , 2018, IEEE Transactions on Software Engineering.

[18]  Yan Wang,et al.  Static Control-Flow Analysis of User-Driven Callbacks in Android Applications , 2015, 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering.

[19]  David Grove,et al.  Optimization of Object-Oriented Programs Using Static Class Hierarchy Analysis , 1995, ECOOP.

[20]  Alessandra Gorla,et al.  Checking app behavior against app descriptions , 2014, ICSE.

[21]  Yifei Zhang,et al.  Launch-Mode-Aware Context-Sensitive Activity Transition Analysis , 2018, 2018 IEEE/ACM 40th International Conference on Software Engineering (ICSE).

[22]  Yan Wang,et al.  Static window transition graphs for Android , 2018, Automated Software Engineering.

[23]  David F. Bacon,et al.  Fast static analysis of C++ virtual function calls , 1996, OOPSLA '96.

[24]  Iulian Neamtiu,et al.  Targeted and depth-first exploration for systematic testing of android apps , 2013, OOPSLA.

[25]  Atanas Rountev,et al.  Static Reference Analysis for GUI Objects in Android Software , 2014, CGO '14.

[26]  Laurie Hendren,et al.  Soot: a Java bytecode optimization framework , 2010, CASCON.

[27]  Eric Bodden,et al.  Boomerang: Demand-Driven Flow- and Context-Sensitive Pointer Analysis for Java , 2016, ECOOP.

[28]  Christopher Krügel,et al.  EdgeMiner: Automatically Detecting Implicit Control Flow Transitions through the Android Framework , 2015, NDSS.

[29]  Alireza Sahami Shirazi,et al.  Insights into layout patterns of mobile user interfaces by an automatic analysis of android apps , 2013, EICS '13.

[30]  Jacques Klein,et al.  FlowDroid: precise context, flow, field, object-sensitive and lifecycle-aware taint analysis for Android apps , 2014, PLDI.