QuickREST: Property-based Test Generation of OpenAPI-Described RESTful APIs

RESTful APIs are an increasingly common way to expose software systems functionality and it is therefore of high interest to find methods to automatically test and verify such APIs. To lower the barrier for industry adoption, such methods need to be straightforward to use with a low effort. This paper introduces a method to explore the behaviour of a RESTful API. This is done by using automatic property-based tests produced from OpenAPI documents that describe the REST API under test. We describe how this method creates artifacts that can be leveraged both as property-based test generators and as a source of validation for results (i.e., as test oracles). Experimental results, on both industrial and open source services, indicate how this approach is a low effort way of finding real faults. Furthermore, it supports building additional knowledge about the system under test by automatically exposing misalignment of specification and implementation. Since the tests are generated from the OpenAPI document this method automatically evolves test cases as the REST API evolves.

[1]  Vaggelis Atlidakis,et al.  RESTler: Stateful REST API Fuzzing , 2019, 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE).

[2]  Robert Feldt Do System Test Cases Grow Old? , 2014, 2014 IEEE Seventh International Conference on Software Testing, Verification and Validation.

[3]  Macías López,et al.  Turning web services descriptions into quickcheck models for automatic testing , 2013, Erlang '13.

[4]  Bernhard K. Aichernig,et al.  Property-based testing of web services by deriving properties from business-rule models , 2017, Software & Systems Modeling.

[5]  Roy Fielding,et al.  Architectural Styles and the Design of Network-based Software Architectures"; Doctoral dissertation , 2000 .

[6]  Konstantinos Sagonas,et al.  A PropEr integration of types and function specifications with property-based testing , 2011, Erlang Workshop.

[7]  Ulf Norell,et al.  Testing AUTOSAR software with QuickCheck , 2015, 2015 IEEE Eighth International Conference on Software Testing, Verification and Validation Workshops (ICSTW).

[8]  Andrea Arcuri,et al.  RESTful API Automated Test Case Generation with EvoMaster , 2019, ACM Trans. Softw. Eng. Methodol..

[9]  Ulf Norell,et al.  Mysteries of DropBox: Property-Based Testing of a Distributed Synchronization Service , 2016, 2016 IEEE International Conference on Software Testing, Verification and Validation (ICST).

[10]  Jordi Cabot,et al.  Automatic Generation of Test Cases for REST APIs: A Specification-Based Approach , 2018, 2018 IEEE 22nd International Enterprise Distributed Object Computing Conference (EDOC).

[11]  Koen Claessen,et al.  QuickCheck: a lightweight tool for random testing of Haskell programs , 2011, SIGP.

[12]  Lars-Åke Fredlund,et al.  Jsongen: a quickcheck based library for testing JSON web services , 2014, Erlang Workshop.

[13]  Lars-Åke Fredlund,et al.  Property-Based Testing of JSON Based Web Services , 2014, 2014 IEEE International Conference on Web Services.

[14]  John Hughes,et al.  Testing telecoms software with quviq QuickCheck , 2006, ERLANG '06.

[15]  John Hughes,et al.  Experiences with QuickCheck: Testing the Hard Stuff and Staying Sane , 2016, A List of Successes That Can Change the World.

[16]  Leonidas Lampropoulos,et al.  Automatic WSDL-guided Test Case Generation for PropEr Testing of Web Services , 2012, WWV.

[17]  Huiqing Li,et al.  Automating property-based testing of evolving web services , 2014, PEPM '14.

[18]  Alberto Lluch-Lafuente,et al.  Microservices: Yesterday, Today, and Tomorrow , 2016, Present and Ulterior Software Engineering.