Detecting Assumptions on Deterministic Implementations of Non-deterministic Specifications

Some commonly used methods have nondeterministicspecifications, e.g., iterating through a set canreturn the elements in any order. However, non-deterministicspecifications typically have deterministic implementations, e.g.,iterating through two sets constructed in the same way mayreturn their elements in the same order. We use the termADINS code to refer to code that Assumes a DeterministicImplementation of a method with a Non-deterministic Specification. Such ADINS code can behave unexpectedly whenthe implementation changes, even if the specification remainsthe same. Further, ADINS code can lead to flaky tests -- teststhat pass or fail seemingly non-deterministically. We present a simple technique, called NONDEX, for detectingflaky tests due to ADINS code. We implemented NONDEX forJava: we found 31 methods with non-deterministic specificationsin the Java Standard Library, manually built non-deterministicmodels for these methods, and used a modified Java VirtualMachine to explore various non-deterministic choices. We evaluatedNONDEX on 195 open-source projects from GitHub and 72student submissions from a programming homework assignment.NONDEX detected 60 flaky tests in 21 open-source projects and110 flaky tests in 34 student submissions.

[1]  Darko Marinov,et al.  Comparing and combining test-suite reduction and regression test selection , 2015, ESEC/SIGSOFT FSE.

[2]  Ralph Johnson,et al.  design patterns elements of reusable object oriented software , 2019 .

[3]  Michael D. Ernst,et al.  Empirically revisiting the test independence assumption , 2014, ISSTA 2014.

[4]  G. Bochmann,et al.  Testing deterministic implementations from nondeterministic FSM specifications , 1996 .

[5]  Nachiappan Nagappan,et al.  Empirically Detecting False Test Alarms Using Association Rules , 2015, 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering.

[6]  Eric Koskinen,et al.  Reasoning about nondeterminism in programs , 2013, PLDI 2013.

[7]  Yuri Gurevich,et al.  Explicating SDKs: Uncovering Assumptions Underlying Secure Authentication and Authorization , 2013, USENIX Security Symposium.

[8]  Wenguang Chen,et al.  Nondeterminism in MapReduce considered harmful? an empirical study on non-commutative aggregators in MapReduce programs , 2014, ICSE Companion.

[9]  Darko Marinov,et al.  An empirical analysis of flaky tests , 2014, SIGSOFT FSE.

[10]  Darko Marinov,et al.  Reliable testing: detecting state-polluting tests to prevent test dependency , 2015, ISSTA.

[11]  Grigore Rosu,et al.  Evolution-Aware Monitoring-Oriented Programming , 2015, 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering.

[12]  Michael D. Ernst,et al.  Feedback-Directed Random Test Generation , 2007, 29th International Conference on Software Engineering (ICSE'07).

[13]  Nina Yevtushenko,et al.  Nondeterministic State Machines in Protocol Conformance Testing , 1993, Protocol Test Systems.

[14]  Tomas Kalibera,et al.  Efficient Detection of Errors in Java Components Using Random Environment and Restarts , 2010, TACAS.

[15]  Chen Huo,et al.  Improving oracle quality by detecting brittle assertions and unused inputs in tests , 2014, FSE 2014.

[16]  Yosi Ben-Asher,et al.  Heuristics for finding concurrent bugs , 2003, Proceedings International Parallel and Distributed Processing Symposium.

[17]  Sarfraz Khurshid,et al.  Test generation through programming in UDITA , 2010, 2010 ACM/IEEE 32nd International Conference on Software Engineering.

[18]  Klaus Havelund,et al.  Model Checking Programs , 2004, Automated Software Engineering.

[19]  Shan Lu,et al.  Understanding and detecting real-world performance bugs , 2012, PLDI.

[20]  R.E. Seviora,et al.  Supervisors for testing non-deterministically specified systems , 1997, Proceedings International Test Conference 1997.

[21]  Mark Harman,et al.  Testing conformance of a deterministic implementation against a non-deterministic stream X-machine , 2004, Theor. Comput. Sci..

[22]  Gail E. Kaiser,et al.  Efficient dependency detection for safe Java test acceleration , 2015, ESEC/SIGSOFT FSE.

[23]  Myra B. Cohen,et al.  Automated testing of GUI applications: Models, tools, and controlling flakiness , 2013, 2013 35th International Conference on Software Engineering (ICSE).

[24]  Gail E. Kaiser,et al.  Unit test virtualization with VMVM , 2014, ICSE.

[25]  Koushik Sen,et al.  A randomized dynamic program analysis technique for detecting real deadlocks , 2009, PLDI '09.

[26]  Michael D. Ernst,et al.  When Tests Collide: Evaluating and Coping with the Impact of Test Dependence , 2015 .