A systematic mapping study of mobile application testing techniques

We performed a systematic mapping study on mobile application testing techniques.79 empirical studies are mapped to a classification schema.Classification scheme includes, structure the topic, evaluation type and contribution facets.Several research gaps in current literature are revealed and discussed. The importance of mobile application specific testing techniques and methods has been attracting much attention of software engineers over the past few years. This is due to the fact that mobile applications are different than traditional web and desktop applications, and more and more they are moving to being used in critical domains. Mobile applications require a different approach to application quality and dependability and require an effective testing approach to build high quality and more reliable software. We performed a systematic mapping study to categorize and to structure the research evidence that has been published in the area of mobile application testing techniques and challenges that they have reported. Seventy nine (79) empirical studies are mapped to a classification schema. Several research gaps are identified and specific key testing issues for practitioners are identified: there is a need for eliciting testing requirements early during development process; the need to conduct research in real-world development environments; specific testing techniques targeting application life-cycle conformance and mobile services testing; and comparative studies for security and usability testing.

[1]  Roel Wieringa,et al.  Requirements engineering paper classification and evaluation criteria: a proposal and a discussion , 2005, Requirements Engineering.

[2]  Caspar Ryan,et al.  Software, performance and resource utilisation metrics for context-aware mobile applications , 2005, 11th IEEE International Software Metrics Symposium (METRICS'05).

[3]  Rachel Harrison,et al.  Lessons Learned from Evaluating the Usability of Mobile Spreadsheet Applications , 2012, HCSE.

[4]  Alton Yeow-Kuan Chua,et al.  Investigating the usability of a mobile location-based annotation system , 2010, MoMM.

[5]  Zhaohui Wang,et al.  Exposing software security and availability risks for commercial mobile devices , 2013, 2013 Proceedings Annual Reliability and Maintainability Symposium (RAMS).

[6]  Wei-Tek Tsai,et al.  Generating Test Cases for Context-Aware Applications Using Bigraphs , 2014, 2014 Eighth International Conference on Software Security and Reliability.

[7]  Luís Carriço,et al.  Lessons from early stages design of mobile applications , 2008, Mobile HCI.

[8]  Vassilis-Javed Khan,et al.  On the use of virtual environments for the evaluation of location-based applications , 2012, CHI.

[9]  Étienne Payet,et al.  Static analysis of Android programs , 2012, Inf. Softw. Technol..

[10]  Ying Huang,et al.  Activity Page Based Functional Test Automation for Android Application , 2012, 2012 Third World Congress on Software Engineering.

[11]  Mark A. Neerincx,et al.  Usability engineering for mobile maps , 2007, Mobility '07.

[12]  Heinrich Hussmann,et al.  A Methodology and Framework to Simplify Usability Analysis of Mobile Applications , 2009, 2009 IEEE/ACM International Conference on Automated Software Engineering.

[13]  Fabio Massacci,et al.  Towards Black Box Testing of Android Apps , 2015, 2015 10th International Conference on Availability, Reliability and Security.

[14]  Shen Yi,et al.  Android Malware Detection Technology Based on Improved Bayesian Classification , 2013, 2013 Third International Conference on Instrumentation, Measurement, Computer, Communication and Control.

[15]  Anders P. Ravn,et al.  Proceedings of the 2006 international workshop on Automation of software test , 2006, International Conference on Software Engineering.

[16]  Wei-Tek Tsai,et al.  Mobile Application Testing: A Tutorial , 2014, Computer.

[17]  Vaninha Vieira,et al.  A context simulator as testing support for mobile apps , 2015, SAC.

[18]  Gerardo Canfora,et al.  A Case Study of Automating User Experience-Oriented Performance Testing on Smartphones , 2013, 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation.

[19]  Qin Cheng,et al.  Usability study of the mobile library App: an example from Chongqing University , 2015, Libr. Hi Tech.

[20]  Porfirio Tramontana,et al.  Considering Context Events in Event-Based Testing of Mobile Applications , 2013, 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation Workshops.

[21]  Marco Pistoia,et al.  Dynamic detection of inter-application communication vulnerabilities in Android , 2015, ISSTA.

[22]  Xiang Long,et al.  Adaptive random testing of mobile application , 2010, 2010 2nd International Conference on Computer Engineering and Technology.

[23]  George C. Necula,et al.  Guided GUI testing of android apps with minimal restart and approximate learning , 2013, OOPSLA.

[24]  Sergiy Vilkomir,et al.  Using Combinatorial Approaches for Testing Mobile Applications , 2014, 2014 IEEE Seventh International Conference on Software Testing, Verification and Validation Workshops.

[25]  Begoña Losada,et al.  Combining InterMod agile methodology with usability engineering in a mobile application development , 2012 .

[26]  Tao Zhang,et al.  Testing Location-Based Function Services for Mobile Applications , 2015, 2015 IEEE Symposium on Service-Oriented System Engineering.

[27]  Ian Warren,et al.  Hermes: A Tool for Testing Mobile Device Applications , 2009, 2009 Australian Software Engineering Conference.

[28]  Bernd Bruegge,et al.  Object-Oriented Software Engineering Using UML, Patterns, and Java , 2009 .

[29]  Ying Zeng,et al.  An automated testing approach for inter-application security in Android , 2014, AST 2014.

[30]  Stephen Kimani,et al.  Appropriating and assessing heuristics for mobile computing , 2006, AVI '06.

[31]  Bin Liu,et al.  SOA based mobile application software test framework , 2009, 2009 8th International Conference on Reliability, Maintainability and Safety.

[32]  Rosziati Ibrahim,et al.  An Adaptation Model for Android Application Testing with Refactoring , 2015 .

[33]  Christian Quesada-López,et al.  Automated Testing of Mobile Applications: A Systematic Map and Review , 2015, CIbSE.

[34]  Yongjian Hu,et al.  Versatile yet lightweight record-and-replay for Android , 2015, OOPSLA.

[35]  Christoffer Quist Adamsen,et al.  Systematic execution of Android test suites in adverse conditions , 2015, ISSTA.

[36]  Derek Flood,et al.  Usability of mobile applications: literature review and rationale for a new usability model , 2013, Journal of Interaction Science.

[37]  Mona Masood,et al.  The Usability of Mobile Applications for Pre-schoolers , 2015 .

[38]  Stefan Kowalewski,et al.  Reverse Engineering and Testing Service Life Cycles of Mobile Platforms , 2012, 2012 23rd International Workshop on Database and Expert Systems Applications.

[39]  Brink van der Merwe,et al.  Verifying android applications using Java PathFinder , 2012, ACM SIGSOFT Softw. Eng. Notes.

[40]  Mariano Ceccato,et al.  Security testing of the communication among Android applications , 2013, 2013 8th International Workshop on Automation of Software Test (AST).

[41]  Ali Shahrokni,et al.  A systematic review of software robustness , 2013, Inf. Softw. Technol..

[42]  Aniruddha S. Gokhale,et al.  Automating testing of service-oriented mobile applications with distributed knowledge and reasoning , 2011, 2011 IEEE International Conference on Service-Oriented Computing and Applications (SOCA).

[43]  Bo Jiang,et al.  FLOMA: Statistical fault localization for mobile embedded system , 2011, 2011 3rd International Conference on Advanced Computer Control.

[44]  Arilo Claudio Dias-Neto,et al.  Automated Mobile Testing as a Service (AM-TaaS) , 2015, 2015 IEEE World Congress on Services.

[45]  Jiang Bo,et al.  MobileTest: A Tool Supporting Automatic Black Box Test for Software on Smart Mobile Devices , 2007, Second International Workshop on Automation of Software Test (AST '07).

[46]  Pekka Jäppinen,et al.  Usability study of ME2.0 , 2011, Personal and Ubiquitous Computing.

[47]  Porfirio Tramontana,et al.  Using GUI ripping for automated testing of Android applications , 2012, 2012 Proceedings of the 27th IEEE/ACM International Conference on Automated Software Engineering.

[48]  Kai Petersen,et al.  Systematic Mapping Studies in Software Engineering , 2008, EASE.

[49]  Timo Ojala,et al.  Testdroid: automated remote UI testing on Android , 2012, MUM.

[50]  Tao Zhang,et al.  Compatibility Testing Service for Mobile Applications , 2015, 2015 IEEE Symposium on Service-Oriented System Engineering.

[51]  Adam A. Porter,et al.  Empirical studies of software engineering: a roadmap , 2000, ICSE '00.

[52]  Henry Muccini,et al.  Software testing of mobile applications: Challenges and future research directions , 2012, 2012 7th International Workshop on Automation of Software Test (AST).

[53]  Volker Gruhn,et al.  Towards Automated UI-Tests for Sensor-Based Mobile Applications , 2015, SoMeT.

[54]  Philippe Kruchten,et al.  Real Challenges in Mobile App Development , 2013, 2013 ACM / IEEE International Symposium on Empirical Software Engineering and Measurement.

[55]  Sebastian G. Elbaum,et al.  Amplifying Tests to Validate Exception Handling Code: An Extended Study in the Mobile Application Domain , 2014, TSEM.

[56]  Iulian Neamtiu,et al.  Targeted and depth-first exploration for systematic testing of android apps , 2013, OOPSLA.

[57]  Ranveer Chandra,et al.  Caiipa: automated large-scale mobile app testing through contextual fuzzing , 2014, MobiCom.

[58]  Oleksii Starov Cloud Platform for Research Crowdsourcing in Mobile Testing , 2013 .

[59]  Woo Jin Lee,et al.  Modeling and Test Case Generation of Inter-component Communication in Android , 2015, 2015 2nd ACM International Conference on Mobile Software Engineering and Systems.

[60]  Siu-Ming Yiu,et al.  DroidChecker: analyzing android applications for capability leak , 2012, WISEC '12.

[61]  Liu Bin,et al.  Test automation on mobile device , 2010, AST.

[62]  A. Jefferson Offutt,et al.  Combination testing strategies: a survey , 2005, Softw. Test. Verification Reliab..

[63]  Sergiy Vilkomir,et al.  Effectiveness of Multi-device Testing Mobile Applications , 2015, 2015 2nd ACM International Conference on Mobile Software Engineering and Systems.

[64]  Lars Nyre,et al.  Design and Evaluation of a Location-Based Mobile News Reader , 2011, 2011 4th IFIP International Conference on New Technologies, Mobility and Security.

[65]  David S. Rosenblum,et al.  Context-Aware Adaptive Applications: Fault Patterns and Their Automated Identification , 2010, IEEE Transactions on Software Engineering.

[66]  Pearl Brereton,et al.  Performing systematic literature reviews in software engineering , 2006, ICSE.

[67]  Miguel Nabuco,et al.  Pattern Based GUI Testing for Mobile Applications , 2014, 2014 9th International Conference on the Quality of Information and Communications Technology.

[68]  Vijayaratnam Ganeshkumar,et al.  A Language Independent Platform for High Secured Communication Using Encrypted Steganography , 2012 .

[69]  Pearl Brereton,et al.  Evidence relating to Object-Oriented software design: A survey , 2007, ESEM 2007.

[70]  Majlinda Fetaji,et al.  Usability testing and evaluation of a mobile software solution: A case study , 2008, ITI 2008 - 30th International Conference on Information Technology Interfaces.

[71]  João Pascoal Faria,et al.  Automated Pattern-Based Testing of Mobile Applications , 2014, 2014 9th International Conference on the Quality of Information and Communications Technology.

[72]  L. Nagowah,et al.  A novel approach of automation testing on mobile devices , 2012, 2012 International Conference on Computer & Information Science (ICCIS).

[73]  Heli Väätäjä,et al.  Bottlenecks, usability issues and development needs in creating and delivering news videos with smart phones , 2010, MoViD '10.

[74]  Robert L. Glass,et al.  Science and substance: a challenge to software engineers , 1994, IEEE Software.

[75]  V. Braun,et al.  Using thematic analysis in psychology , 2006 .

[76]  Sergiy Vilkomir,et al.  Testing-as-a-Service for Mobile Applications: State-of-the-Art Survey , 2015 .

[77]  Marko Palola,et al.  Towards automating testing of communicational B3G applications , 2006, Mobility '06.

[78]  Jin Hyung Kim,et al.  An Integrated Test Automation Framework for Testing on Heterogeneous Mobile Platforms , 2011, 2011 First ACIS International Symposium on Software and Network Engineering.

[79]  Stefan Kowalewski,et al.  Testing Conformance of Life Cycle Dependent Properties of Mobile Applications , 2012, 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation.

[80]  Byoungju Choi,et al.  Performance Testing of Mobile Applications at the Unit Test Level , 2009, 2009 Third IEEE International Conference on Secure Software Integration and Reliability Improvement.

[81]  Porfirio Tramontana,et al.  MobiGUITAR: Automated Model-Based Testing of Mobile Apps , 2015, IEEE Software.

[82]  Porfirio Tramontana,et al.  A GUI Crawling-Based Technique for Android Mobile Application Testing , 2011, 2011 IEEE Fourth International Conference on Software Testing, Verification and Validation Workshops.

[83]  Bernd Bruegge,et al.  Object-Oriented Software Engineering: Using UML, Patterns and Java, Second Edition , 2003 .

[84]  Sébastien Salva,et al.  Data vulnerability detection by security testing for Android applications , 2013, 2013 Information Security for South Africa.

[85]  P. Vincent,et al.  Observations on performance of client-server mobile applications , 2008, 2008 1st International Conference on Information Technology.

[86]  Tiziana Catarci,et al.  A unified methodology for the evaluation of accessibility and usability of mobile applications , 2010, Universal Access in the Information Society.

[87]  Lisa Crispin,et al.  Agile Testing: A Practical Guide for Testers and Agile Teams , 2008 .

[88]  Ratul Mahajan,et al.  AppInsight: Mobile App Performance Monitoring in the Wild , 2022 .

[89]  Raymond A. Paul,et al.  Mobile Testing-as-a-Service (MTaaS) -- Infrastructures, Issues, Solutions and Needs , 2014, 2014 IEEE 15th International Symposium on High-Assurance Systems Engineering.

[90]  Jerry Zeyu Gao,et al.  Modeling mobile application test platform and environment: testing criteria and complexity analysis , 2014, JAMAICA 2014.

[91]  Jeff Yu Lei,et al.  Practical Combinatorial Testing: Beyond Pairwise , 2008, IT Professional.

[92]  Hsiang-Lin Wen,et al.  PATS: A Parallel GUI Testing Framework for Android Applications , 2015, 2015 IEEE 39th Annual Computer Software and Applications Conference.

[93]  Junfeng Yang,et al.  Efficiently, effectively detecting mobile app bugs with AppDoctor , 2014, EuroSys '14.

[94]  Marek Milosz,et al.  Mobile application usability testing in quasi-real conditions , 2015, 2015 8th International Conference on Human System Interaction (HSI).

[95]  Porfirio Tramontana,et al.  AGRippin: a novel search based testing technique for Android applications , 2015, DeMobile@SIGSOFT FSE.

[96]  André Takeshi Endo,et al.  Evaluating the Model-Based Testing Approach in the Context of Mobile Applications , 2015, CLEI Selected Papers.

[97]  Volker Gruhn,et al.  xploring the benefits of the combination of a software architecture nalysis and a usability evaluation of a mobile application , 2010 .

[98]  Vaninha Vieira,et al.  Smartphone Applications Usability Evaluation: A Hybrid Model and Its Implementation , 2012, HCSE.

[99]  David S. Rosenblum,et al.  Automated Generation of Context-Aware Tests , 2007, 29th International Conference on Software Engineering (ICSE'07).

[100]  Auri Marcelo Rizzo Vincenzi,et al.  A strategy to perform coverage testing of mobile applications , 2006, AST '06.

[101]  David Aspinall,et al.  Security testing for Android mHealth apps , 2015, 2015 IEEE Eighth International Conference on Software Testing, Verification and Validation Workshops (ICSTW).

[102]  Clemens Holzmann,et al.  Automated and unsupervised user interaction logging as basis for usability evaluation of mobile applications , 2012, MoMM '12.

[103]  Zhimin Wang Validating context-aware applications , 2008 .