Analysing the Program Analyser

The reliability of program analysis tools is clearly important if such tools are to play a serious role in improving the quality and integrity of software systems, and the confidence which users place in such systems. Yet our experience is that, currently, little attention is paid to analysing the correctness of program analysers themselves, beyond regression testing. In this position paper we present our vision that, by 2025, the use of more rigorous analyses to check the reliability of program analysers will be commonplace. Inspired by recent advances in compiler testing, we set out initial steps towards this vision, building upon techniques such as cross-checking, program transformation and program generation.

[1]  Armin Biere,et al.  Fuzzing and delta-debugging SMT solvers , 2009, SMT '09.

[2]  W. M. McKeeman,et al.  Differential Testing for Software , 1998, Digit. Tech. J..

[3]  Xavier Leroy,et al.  Formal verification of a realistic compiler , 2009, CACM.

[4]  Dawson R. Engler,et al.  KLEE: Unassisted and Automatic Generation of High-Coverage Tests for Complex Systems Programs , 2008, OSDI.

[5]  Zhendong Su,et al.  Compiler validation via equivalence modulo inputs , 2014, PLDI.

[6]  Chanchal Kumar Roy,et al.  A Mutation/Injection-Based Automatic Framework for Evaluating Code Clone Detection Tools , 2009, 2009 International Conference on Software Testing, Verification, and Validation Workshops.

[7]  John Wickerson,et al.  The Design and Implementation of a Verification Technique for GPU Kernels , 2015, TOPL.

[8]  Flash Sheridan,et al.  Practical testing of a C99 compiler using output comparison , 2007, Softw. Pract. Exp..

[9]  Amir Pnueli,et al.  Translation Validation , 1998, TACAS.

[10]  Elaine J. Weyuker,et al.  On Testing Non-Testable Programs , 1982, Comput. J..

[11]  Mark Harman,et al.  An Analysis and Survey of the Development of Mutation Testing , 2011, IEEE Transactions on Software Engineering.

[12]  Darko Marinov,et al.  Automated testing of refactoring engines , 2007, ESEC-FSE '07.

[13]  Xuejun Yang,et al.  Finding and understanding bugs in C compilers , 2011, PLDI '11.

[14]  Robert N. M. Watson,et al.  Into the depths of C: elaborating the de facto standards , 2016, PLDI.

[15]  Cristian Cadar,et al.  Targeted program transformations for symbolic execution , 2015, ESEC/SIGSOFT FSE.

[16]  Junfeng Yang,et al.  Effective dynamic detection of alias analysis errors , 2013, ESEC/FSE 2013.

[17]  Dirk Beyer.  Software Verification and Verifiable Witnesses - (Report on SV-COMP 2015) , 2015, TACAS.

[18]  References , 1971 .

[19]  Alastair F. Donaldson,et al.  Many-core compiler fuzzing , 2015, PLDI.

[20]  Elaine J. Weyuker,et al.  Pseudo-oracles for non-testable programs , 1981, ACM '81.