Why My App Crashes? Understanding and Benchmarking Framework-Specific Exceptions of Android Apps

Mobile apps have become ubiquitous. Ensuring their correctness and reliability is important. However, many apps still suffer from occasional to frequent crashes, weakening their competitive edge. Large-scale, deep analyses of the characteristics of real-world app crashes can provide useful insights to both developers and researchers. However, such studies are difficult and yet to be carried out — this work fills this gap. We collected 16,245 and 8,760 unique exceptions from 2,486 open-source and 3,230 commercial Android apps, respectively, and observed that the exceptions thrown from Android framework (termed “framework-specific exceptions” ) account for the majority. With one-year effort, we (1) extensively investigated these framework-specific exceptions, and (2) further conducted an online survey of 135 professional app developers about how they analyze, test, reproduce and fix these exceptions. Specifically, we aim to understand the framework-specific exceptions from several perspectives: (i) their characteristics (e.g., manifestation locations, fault taxonomy), (ii) the developers’ testing practices, (iii) existing bug detection techniques’ effectiveness, (iv) their reproducibility and (v) bug fixes. To enable follow-up research (e.g., bug understanding, detection, localization and repairing), we further systematically constructed, DroidDefects, the first comprehensive and largest benchmark of Android app exception bugs. This benchmark contains 33 reproducible exceptions (with test cases, stack traces, faulty and fixed app versions, bug types, etc.), and 3,696 ground-truth exceptions (real faults manifested by automated testing tools), which cover the apps with different complexities and diverse exception types. Based on our findings, we also built two prototype tools: Stoat+, an optimized dynamic testing tool, which quickly uncovered three previously-unknown, fixed crashes in Gmail and Google+; ExLocator, an exception localization tool, which can locate the root causes of specific exception types. Our dataset, benchmark and tools are publicly available on https:// github.com/ tingsu/ droiddefects.

[1]  Iulian Neamtiu,et al.  Finding resume and restart errors in Android applications , 2016, OOPSLA.

[2]  Yang Liu,et al.  Efficiently Manifesting Asynchronous Programming Errors in Android Apps , 2018, 2018 33rd IEEE/ACM International Conference on Automated Software Engineering (ASE).

[3]  Ana C. R. Paiva,et al.  Why does the orientation change mess up my Android application? From GUI failures to code faults , 2018, Softw. Test. Verification Reliab..

[4]  Arie van Deursen,et al.  Unveiling Exception Handling Bug Hazards in Android Based on GitHub and Google Code Issues , 2015, MSR.

[5]  Hossain Shahriar,et al.  Testing of Memory Leak in Android Applications , 2014, 2014 IEEE 15th International Symposium on High-Assurance Systems Engineering.

[6]  Miryung Kim,et al.  An Empirical Study of API Stability and Adoption in the Android Ecosystem , 2013, 2013 IEEE International Conference on Software Maintenance.

[7]  A. Jefferson Offutt,et al.  Mutation operators for testing Android apps , 2017, Inf. Softw. Technol..

[8]  Brink van der Merwe,et al.  Verifying android applications using Java PathFinder , 2012, ACM SIGSOFT Softw. Eng. Notes.

[9]  Gabriel Nunez,et al.  Party Pooper: Third-Party Libraries in Android , 2011 .

[10]  Christopher Vendome,et al.  How developers detect and fix performance bottlenecks in Android apps , 2015, 2015 IEEE International Conference on Software Maintenance and Evolution (ICSME).

[11]  Yepang Liu,et al.  A Tale of Two Cities: How WebView Induces Bugs to Android Applications , 2018, 2018 33rd IEEE/ACM International Conference on Automated Software Engineering (ASE).

[12]  Yang Liu,et al.  ReCDroid: Automatically Reproducing Android Application Crashes from Bug Reports , 2019, 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE).

[13]  Yepang Liu,et al.  Understanding and detecting wake lock misuses for Android applications , 2016, SIGSOFT FSE.

[14]  Darko Marinov,et al.  An empirical analysis of flaky tests , 2014, SIGSOFT FSE.

[15]  Lilly Irani,et al.  Amazon Mechanical Turk , 2018, Advances in Intelligent Systems and Computing.

[16]  Jian Zhang,et al.  DroidLeaks: a comprehensive database of resource leaks in Android apps , 2019, Empirical Software Engineering.

[17]  Daniela Micucci,et al.  From source code to test cases: A comprehensive benchmark for resource leak detection in Android apps , 2018, Softw. Pract. Exp..

[18]  Lingling Fan,et al.  Model-Based Continuous Verification , 2016, 2016 23rd Asia-Pacific Software Engineering Conference (APSEC).

[19]  Mario Linares Vásquez,et al.  Auto-completing bug reports for Android applications , 2015, ESEC/SIGSOFT FSE.

[20]  Binyu Zang,et al.  Espresso , 2018, ACM SIGPLAN Notices.

[21]  Alexander Aiken,et al.  Minimizing GUI event traces , 2016, SIGSOFT FSE.

[22]  Satish Narayanasamy,et al.  Race detection for event-driven mobile applications , 2014, PLDI.

[23]  Na Meng,et al.  An Empirical Study of Flaky Tests in Android Apps , 2018, 2018 IEEE International Conference on Software Maintenance and Evolution (ICSME).

[24]  Camilo Gómez-Rivas,et al.  Fragments , 2013 .

[25]  Zhendong Su,et al.  A Survey on Data-Flow Testing , 2017, ACM Comput. Surv..

[26]  Laurie Hendren,et al.  Soot: a Java bytecode optimization framework , 2010, CASCON.

[27]  Bo Jiang,et al.  SimplyDroid: Efficient event sequence simplification for android application , 2017, 2017 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE).

[28]  A. Jefferson Offutt,et al.  Towards mutation analysis of Android apps , 2015, 2015 IEEE Eighth International Conference on Software Testing, Verification and Validation Workshops (ICSTW).

[29]  Mayur Naik,et al.  Dynodroid: an input generation system for Android apps , 2013, ESEC/FSE 2013.

[30]  Yang Liu,et al.  Large-Scale Analysis of Framework-Specific Exceptions in Android Apps , 2018, 2018 IEEE/ACM 40th International Conference on Software Engineering (ICSE).

[31]  Lingling Fan,et al.  StoryDroid: Automated Generation of Storyboard for Android Apps , 2019, 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE).

[32]  J Salvage A root cause? , 1982, Nursing times.

[33]  Hai-Feng Guo,et al.  Debugging Energy-Efficiency Related Field Failures in Mobile Apps , 2016, 2016 IEEE/ACM International Conference on Mobile Software Engineering and Systems (MOBILESoft).

[34]  Iulian Neamtiu,et al.  Targeted and depth-first exploration for systematic testing of android apps , 2013, OOPSLA.

[35]  Yepang Liu,et al.  Understanding and Detecting Callback Compatibility Issues for Android Applications , 2018, 2018 33rd IEEE/ACM International Conference on Automated Software Engineering (ASE).

[36]  Lionel C. Briand,et al.  A Hitchhiker's guide to statistical tests for assessing randomized algorithms in software engineering , 2014, Softw. Test. Verification Reliab..

[37]  Xiang Gao,et al.  Repairing Crashes in Android Apps , 2018, 2018 IEEE/ACM 40th International Conference on Software Engineering (ICSE).

[38]  Tao Xie,et al.  A Grey-Box Approach for Automated GUI-Model Generation of Mobile Applications , 2013, FASE.

[39]  Sarfraz Khurshid,et al.  Automated Generation of Oracles for Testing User-Interaction Features of Mobile Apps , 2014, 2014 IEEE Seventh International Conference on Software Testing, Verification and Validation.

[40]  Porfirio Tramontana,et al.  Using GUI ripping for automated testing of Android applications , 2012, 2012 Proceedings of the 27th IEEE/ACM International Conference on Automated Software Engineering.

[41]  Lingling Fan,et al.  A Large-Scale Empirical Study on Industrial Fake Apps , 2019, 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP).

[42]  Suman Nath,et al.  PUMA: programmable UI-automation for large-scale dynamic analysis of mobile apps , 2014, MobiSys.

[43]  Yuriy Brun,et al.  The ManyBugs and IntroClass Benchmarks for Automated Repair of C Programs , 2015, IEEE Transactions on Software Engineering.

[44]  Tao Xie,et al.  An Empirical Study of Android Test Generation Tools in Industrial Cases , 2018, 2018 33rd IEEE/ACM International Conference on Automated Software Engineering (ASE).

[45]  Shujuan Jiang,et al.  Fault localization and repair for Java runtime exceptions , 2009, ISSTA.

[46]  Mira Mezini,et al.  MUBench: A Benchmark for API-Misuse Detectors , 2016, 2016 IEEE/ACM 13th Working Conference on Mining Software Repositories (MSR).

[47]  George C. Necula,et al.  DetReduce: Minimizing Android GUI Test Suites for Regression Testing , 2018, 2018 IEEE/ACM 40th International Conference on Software Engineering (ICSE).

[48]  Abbas Heydarnoori,et al.  Exception Fault Localization in Android Applications , 2015, 2015 2nd ACM International Conference on Mobile Software Engineering and Systems.

[49]  Christoffer Quist Adamsen,et al.  Systematic execution of Android test suites in adverse conditions , 2015, ISSTA.

[50]  Standard Glossary of Terms used in Software Testing Version 3 . 1 Foundation Terms , .

[51]  Junfeng Yang,et al.  Efficiently, effectively detecting mobile app bugs with AppDoctor , 2014, EuroSys '14.

[52]  Yepang Liu,et al.  Taming Android fragmentation: Characterizing and detecting compatibility issues for Android apps , 2016, 2016 31st IEEE/ACM International Conference on Automated Software Engineering (ASE).

[53]  Alessandra Gorla,et al.  Automated Test Input Generation for Android: Are We There Yet? (E) , 2015, 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE).

[54]  Shujuan Jiang,et al.  A Debugging Approach for Java Runtime Exceptions Based on Program Slicing and Stack Traces , 2010, 2010 10th International Conference on Quality Software.

[55]  Jian Lu,et al.  AimDroid: Activity-Insulated Multi-level Automated Testing for Android Applications , 2017, 2017 IEEE International Conference on Software Maintenance and Evolution (ICSME).

[56]  Ahmed E. Hassan,et al.  What Do Mobile App Users Complain About? , 2015, IEEE Software.

[57]  Erik Derr,et al.  Keep me Updated: An Empirical Study of Third-Party Library Updatability on Android , 2017, CCS.

[58]  Minhui Xue,et al.  AUSERA: Large-Scale Automated Security Risk Assessment of Global Mobile Banking Apps , 2018, ArXiv.

[59]  Hongseok Yang,et al.  Automated concolic testing of smartphone apps , 2012, SIGSOFT FSE.

[60]  Martin T. Vechev,et al.  Scalable race detection for Android applications , 2015, OOPSLA.

[61]  Sam Malek,et al.  EvoDroid: segmented evolutionary testing of Android apps , 2014, SIGSOFT FSE.

[62]  George C. Necula,et al.  Guided GUI testing of android apps with minimal restart and approximate learning , 2013, OOPSLA.

[63]  Xiaoyin Wang,et al.  Experience paper: a study on behavioral backward incompatibilities of Java software libraries , 2017, ISSTA.

[64]  Ting Su,et al.  FSMdroid: Guided GUI Testing of Android Apps , 2016, 2016 IEEE/ACM 38th International Conference on Software Engineering Companion (ICSE-C).

[65]  Michael D. Ernst,et al.  Defects4J: a database of existing faults to enable controlled testing studies for Java programs , 2014, ISSTA 2014.

[66]  S. Malek,et al.  PATDroid: permission-aware GUI testing of Android , 2017, ESEC/SIGSOFT FSE.

[67]  David Lo,et al.  Understanding the Test Automation Culture of App Developers , 2015, 2015 IEEE 8th International Conference on Software Testing, Verification and Validation (ICST).

[68]  Minhui Xue,et al.  GUI-Squatting Attack: Automated Generation of Android Phishing Apps , 2019, IEEE Transactions on Dependable and Secure Computing.

[69]  David Lo,et al.  Predicting Crashing Releases of Mobile Applications , 2016, ESEM.

[70]  Mario Linares Vásquez,et al.  FUSION: A Tool for Facilitating and Augmenting Android Bug Reporting , 2016, 2016 IEEE/ACM 38th International Conference on Software Engineering Companion (ICSE-C).

[71]  Nataniel P. Borges Data flow oriented UI testing: exploiting data flows and UI elements to test Android applications , 2017, ISSTA.

[72]  René Just,et al.  The major mutation framework: efficient and scalable mutation analysis for Java , 2014, ISSTA 2014.

[73]  Iulian Neamtiu,et al.  Automating GUI testing for Android applications , 2011, AST '11.

[74]  Lingling Fan,et al.  Are mobile banking apps secure? what can be improved? , 2018, ESEC/SIGSOFT FSE.

[75]  Gabriele Bavota,et al.  MDroid+: A Mutation Testing Framework for Android , 2018, 2018 IEEE/ACM 40th International Conference on Software Engineering: Companion (ICSE-Companion).

[76]  Myra B. Cohen,et al.  Configurations in Android testing: they matter , 2018, A-Mobile@ASE.

[77]  Yepang Liu,et al.  Understanding and Detecting Fragmentation-Induced Compatibility Issues for Android Apps , 2020, IEEE Transactions on Software Engineering.

[78]  Mario Linares Vásquez,et al.  Generating Reproducible and Replayable Bug Reports from Android Application Crashes , 2015, 2015 IEEE 23rd International Conference on Program Comprehension.

[79]  Swarat Chaudhuri,et al.  A Study of Android Application Security , 2011, USENIX Security Symposium.

[80]  Mark Harman,et al.  An Analysis and Survey of the Development of Mutation Testing , 2011, IEEE Transactions on Software Engineering.

[81]  Aspen Olmsted,et al.  Memory leak detection in android applications based on code patterns , 2016, 2016 International Conference on Information Society (i-Society).

[82]  Porfirio Tramontana,et al.  MobiGUITAR: Automated Model-Based Testing of Mobile Apps , 2015, IEEE Software.

[83]  Laurence Moroney Google Analytics for Firebase , 2017 .

[84]  Jeff Huang,et al.  EHBDroid: Beyond GUI testing for Android applications , 2017, 2017 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE).

[85]  Yue Jia,et al.  Sapienz: multi-objective automated testing for Android applications , 2016, ISSTA.

[86]  Saurabh Bagchi,et al.  Characterizing Failures in Mobile OSes: A Case Study with Android and Symbian , 2010, 2010 IEEE 21st International Symposium on Software Reliability Engineering.

[87]  Myra B. Cohen,et al.  Automated testing of GUI applications: Models, tools, and controlling flakiness , 2013, 2013 35th International Conference on Software Engineering (ICSE).

[88]  Gabriele Bavota,et al.  Enabling mutation testing for Android apps , 2017, ESEC/SIGSOFT FSE.

[89]  Yang Liu,et al.  Guided, stochastic model-based GUI testing of Android apps , 2017, ESEC/SIGSOFT FSE.

[90]  Christopher Vendome,et al.  Automatically Discovering, Reporting and Reproducing Android Application Crashes , 2016, 2016 IEEE International Conference on Software Testing, Verification and Validation (ICST).

[91]  Alessandro Orso,et al.  Automatically translating bug reports into test cases for mobile apps , 2018, ISSTA.

[92]  Rongxin Wu,et al.  CrashLocator: locating crashing faults based on crash stacks , 2014, ISSTA 2014.

[93]  Mario Linares Vásquez,et al.  How do Developers Test Android Applications? , 2017, 2017 IEEE International Conference on Software Maintenance and Evolution (ICSME).

[94]  Christopher Vendome,et al.  CrashScope: A Practical Tool for Automated Testing of Android Applications , 2017, 2017 IEEE/ACM 39th International Conference on Software Engineering Companion (ICSE-C).

[95]  A. Jefferson Offutt,et al.  Is Mutation Analysis Effective at Testing Android Apps? , 2017, 2017 IEEE International Conference on Software Quality, Reliability and Security (QRS).

[96]  Yepang Liu,et al.  Characterizing and detecting performance bugs for smartphone applications , 2014, ICSE.

[97]  Celso G. Camilo-Junior,et al.  DroidBugs: An Android Benchmark for Automatic Program Repair , 2018, ArXiv.