Metamorphic Testing of RESTful Web APIs

Web Application Programming Interfaces (APIs) allow systems to interact with each other over the network. Modern Web APIs often adhere to the REST architectural style, being referred to as RESTful Web APIs. RESTful Web APIs are decomposed into multiple resources (e.g., a video in the YouTube API) that clients can manipulate through HTTP interactions. Testing Web APIs is critical but challenging due to the difficulty to assess the correctness of API responses, i.e., the oracle problem. Metamorphic testing alleviates the oracle problem by exploiting relations (so-called metamorphic relations) among multiple executions of the program under test. In this paper, we present a metamorphic testing approach for the detection of faults in RESTful Web APIs. We first propose six abstract relations that capture the shape of many of the metamorphic relations found in RESTful Web APIs, we call these Metamorphic Relation Output Patterns (MROPs). Each MROP can then be instantiated into one or more concrete metamorphic relations. The approach was evaluated using both automatically seeded and real faults in six subject Web APIs. Among other results, we identified 60 metamorphic relations (instances of the proposed MROPs) in the Web APIs of Spotify and YouTube. Each metamorphic relation was implemented using both random and manual test data, running over 4.7K automated tests. As a result, 11 issues were detected (3 in Spotify and 8 in YouTube), 10 of them confirmed by the API developers or reproduced by other users, supporting the effectiveness of the approach.

[1]  Huai Liu,et al.  An innovative approach for testing bioinformatics programs using metamorphic testing , 2009, BMC Bioinformatics.

[2]  Tsong Yueh Chen,et al.  Metamorphic Testing for Software Quality Assessment: A Study of Search Engines , 2016, IEEE Transactions on Software Engineering.

[3]  Ivan Porres,et al.  Designing level 3 behavioral RESTful web service interfaces , 2011, SIAP.

[4]  Mikael Lindvall,et al.  Metamorphic Model-Based Testing Applied on NASA DAT -- An Experience Report , 2015, 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering.

[5]  A. Jefferson Offutt,et al.  MuJava: an automated class mutation system , 2005, Softw. Test. Verification Reliab..

[6]  Tsong Yueh Chen,et al.  Metamorphic Testing: A New Approach for Generating Next Test Cases , 2020, ArXiv.

[7]  Richard J. Lipton,et al.  Hints on Test Data Selection: Help for the Practicing Programmer , 1978, Computer.

[8]  Bohn Stafleu van Loghum,et al.  Online … , 2002, LOG IN.

[9]  Sergio Segura,et al.  Automated inference of likely metamorphic relations for model transformations , 2018, J. Syst. Softw..

[10]  Gail E. Kaiser,et al.  Dynamic Inference of Likely Metamorphic Properties to Support Differential Testing , 2015, 2015 IEEE/ACM 10th International Workshop on Automation of Software Test.

[11]  Dong Wang,et al.  Extending WSDL to facilitate Web services testing , 2002, 7th IEEE International Symposium on High Assurance Systems Engineering, 2002. Proceedings..

[12]  T. H. Tse,et al.  Finding failures from passed test cases: improving the pattern classification approach to the testing of mesh simplification programs , 2010, Softw. Test. Verification Reliab..

[13]  Raymond A. Paul,et al.  Coyote: an XML-based framework for Web services testing , 2002, 7th IEEE International Symposium on High Assurance Systems Engineering, 2002. Proceedings..

[14]  Subbu Allamaraju.  RESTful Web Services Cookbook , 2010 .

[15]  Gail E. Kaiser,et al.  Properties of Machine Learning Applications for Use in Metamorphic Testing , 2008, SEKE.

[16]  Zhendong Su,et al.  Compiler validation via equivalence modulo inputs , 2014, PLDI.

[17]  Jeffrey M. Voas,et al.  Metamorphic Testing for Cybersecurity , 2016, Computer.

[18]  Tao Zhang,et al.  WSDL-Based Automated Test Data Generation for Web Service , 2008, 2008 International Conference on Computer Science and Software Engineering.

[19]  Huai Liu,et al.  How Effectively Does Metamorphic Testing Alleviate the Oracle Problem? , 2014, IEEE Transactions on Software Engineering.

[20]  Huai Liu,et al.  A Metamorphic Relation-Based Approach to Testing Web Services Without Oracles , 2012, Int. J. Web Serv. Res..

[21]  Savas Parastatidis,et al.  REST in Practice - Hypermedia and Systems Architecture , 2010 .

[22]  W. Chan,et al.  A Metamorphic Testing Approach for Online Testing of Service-Oriented Software Applications , 2007, Int. J. Web Serv. Res..

[23]  Wei-Tek Tsai,et al.  WSDL-based automatic test case generation for Web services testing , 2005, IEEE International Workshop on Service-Oriented System Engineering (SOSE'05).

[24]  Leonard Richardson,et al.  RESTful Web APIs , 2013 .

[25]  James M. Bieman,et al.  Predicting metamorphic relations for testing scientific software: a machine learning approach using graph kernels , 2016, Softw. Test. Verification Reliab..

[26]  Sergio Segura,et al.  A Survey on Metamorphic Testing , 2016, IEEE Transactions on Software Engineering.

[27]  Mark Harman,et al.  An Analysis and Survey of the Development of Mutation Testing , 2011, IEEE Transactions on Software Engineering.

[28]  Tsong Yueh Chen,et al.  Automated functional testing of web search engines in the absence of an oracle , 2007 .

[29]  Tsong Yueh Chen,et al.  METRIC: METamorphic Relation Identification based on the Category-choice framework , 2016, J. Syst. Softw..

[30]  Mark Masse,et al.  REST API Design Rulebook , 2011 .

[31]  Gerardo Canfora,et al.  Service-Oriented Architectures Testing: A Survey , 2009, ISSSE.

[32]  Prashant Kumar,et al.  Test-the-REST: An Approach to Testing RESTful Web-Services , 2009, 2009 Computation World: Future Computing, Service Computation, Cognitive, Adaptive, Content, Patterns.

[33]  Elaine J. Weyuker,et al.  On Testing Non-Testable Programs , 1982, Comput. J..

[34]  Daniel Jacobson,et al.  APIs: A Strategy Guide , 2011 .

[35]  Inmaculada Medina-Bulo,et al.  Application of Metamorphic Testing to a Case Study in Web Services Compositions , 2011, ICETE.

[36]  Tsong Yueh Chen,et al.  Automated functional testing of online search services , 2012, Softw. Test. Verification Reliab..

[37]  Mikael Lindvall,et al.  Agile Metamorphic Model-Based Testing , 2016, 2016 IEEE/ACM 1st International Workshop on Metamorphic Testing (MET).

[38]  Ivan Porres,et al.  Modeling behavioral RESTful web service interfaces in UML , 2011, SAC '11.

[39]  M. Harman,et al.  Testing Web Services : A Survey , 2011 .

[40]  Adenilso Simao,et al.  Model-Based Testing of RESTful Web Services Using UML Protocol State Machines , 2013 .

[41]  Gail E. Kaiser,et al.  Using JML Runtime Assertion Checking to Automate Metamorphic Testing in Applications without Test Oracles , 2009, 2009 International Conference on Software Testing Verification and Validation.

[42]  Mark Harman,et al.  The Oracle Problem in Software Testing: A Survey , 2015, IEEE Transactions on Software Engineering.