FlinkCheck: Property-Based Testing for Apache Flink

Apache Flink is an open-source, soft real-time stream processing framework underlying many modern systems dealing with cloud and real-time computing, data analytics, and the Internet of Things, among others. As the complexity of stream-processing systems increase, the testing, debugging, and verification tools supporting them should improve as well. However, Flink’s testing tools only include a local cluster fake, which requires a great effort from the user to manually craft all those streams (and their corresponding output) that are relevant for each particular function under test. Property-based testing is an automatic, black-box testing technique that tests functions by generating random inputs and checking whether the obtained outputs fulfill a given property. In this paper, we present FlinkCheck, a property-based testing tool for Apache Flink. It uses a bounded temporal logic for both guiding how random streams are generated and defining the properties. We illustrate how the tool works with an example of a collaborative initiative against sexual harassment.

[1]  Martin Odersky,et al.  An Overview of the Scala Programming Language , 2004 .

[2]  Geoff Holmes,et al.  MOA: Massive Online Analysis , 2010, J. Mach. Learn. Res..

[3]  Mark Harman,et al.  An Analysis and Survey of the Development of Mutation Testing , 2011, IEEE Transactions on Software Engineering.

[4]  Tom White,et al.  Hadoop: The Definitive Guide , 2009 .

[5]  Howard Barringer,et al.  TraceContract: A Scala DSL for Trace Analysis , 2011, FM.

[6]  Seif Haridi,et al.  Apache Flink™: Stream and Batch Processing in a Single Engine , 2015, IEEE Data Eng. Bull..

[7]  Rebecca S. Robinson,et al.  Cooperative Networks in Interorganizational Settings: Analyzing Cyber-Collective Action , 2013 .

[8]  Robert M. Hierons,et al.  An extended framework for passive asynchronous testing , 2017, J. Log. Algebraic Methods Program..

[9]  John Hughes,et al.  Testing telecoms software with quviq QuickCheck , 2006, ERLANG '06.

[10]  Konstantinos Sagonas,et al.  Targeted property-based testing , 2017, ISSTA.

[11]  Koushik Sen,et al.  JQF: coverage-guided property-based testing in Java , 2019, ISSTA.

[12]  Scott Shenker,et al.  Discretized streams: fault-tolerant streaming computation at scale , 2013, SOSP.

[13]  Carsten Franke,et al.  MARS: A flexible real-time streaming platform for testing automation systems , 2017, Design, Automation & Test in Europe Conference & Exhibition (DATE), 2017.

[14]  Vítor Santos Costa,et al.  PrologCheck - Property-Based Testing in Prolog , 2014, FLOPS.

[15]  Robert M. Hierons,et al.  A tool supported methodology to passively test asynchronous systems with multiple users , 2018, Inf. Softw. Technol..

[16]  Tiago H. Falk,et al.  Audio-Visual Multimedia Quality Assessment: A Comprehensive Survey , 2017, IEEE Access.

[17]  Pascal Raymond,et al.  Lutin: A Language for Specifying and Executing Reactive Scenarios , 2008, EURASIP J. Embed. Syst..

[18]  Choongwoo Han,et al.  The Art, Science, and Engineering of Fuzzing: A Survey , 2018, IEEE Transactions on Software Engineering.

[19]  Jaeil Kim,et al.  Test stream generation for digital cable UHD broadcasting standard , 2016, 2016 IEEE 6th International Conference on Consumer Electronics - Berlin (ICCE-Berlin).

[20]  Gerard Meszaros,et al.  xUnit Test Patterns: Refactoring Test Code , 2007 .

[21]  Bernd Finkbeiner,et al.  LOLA: runtime monitoring of synchronous systems , 2005, 12th International Symposium on Temporal Representation and Reasoning (TIME'05).

[22]  Yves Le Traon,et al.  Validity Fuzzing and Parametric Generators for Effective Random Testing , 2018, 2019 IEEE/ACM 41st International Conference on Software Engineering: Companion Proceedings (ICSE-Companion).

[23]  Ana R. Cavalli,et al.  New approaches for passive testing using an Extended Finite State Machine specification , 2003, Inf. Softw. Technol..

[24]  Konstantinos Sagonas,et al.  A PropEr integration of types and function specifications with property-based testing , 2011, Erlang Workshop.

[25]  Nicolas Halbwachs,et al.  Synchronous Programming of Reactive Systems , 1992, CAV.

[26]  Yves Le Traon,et al.  Semantic fuzzing with zest , 2018, ISSTA.

[27]  Catherine C. McGeoch A Guide to Experimental Algorithmics , 2012 .

[28]  Gordon Fraser,et al.  Random or Genetic Algorithm Search for Object-Oriented Test Suite Generation? , 2015, GECCO.

[29]  Albert Bifet,et al.  Massive Online Analysis , 2009 .

[30]  Koen Claessen,et al.  QuickCheck: a lightweight tool for random testing of Haskell programs , 2011, SIGP.

[31]  Adrián Riesco,et al.  Property-based testing for Spark Streaming , 2019, Theory Pract. Log. Program..