Taming compiler fuzzers

Aggressive random testing tools ("fuzzers") are impressively effective at finding compiler bugs. For example, a single test-case generator has resulted in more than 1,700 bugs reported for a single JavaScript engine. However, fuzzers can be frustrating to use: they indiscriminately and repeatedly find bugs that may not be severe enough to fix right away. Currently, users filter out undesirable test cases using ad hoc methods such as disallowing problematic features in tests and grepping test results. This paper formulates and addresses the fuzzer taming problem: given a potentially large number of random test cases that trigger failures, order them such that diverse, interesting test cases are highly ranked. Our evaluation shows our ability to solve the fuzzer taming problem for 3,799 test cases triggering 46 bugs in a C compiler and 2,603 test cases triggering 28 bugs in a JavaScript engine.

[1]  Andreas Zeller,et al.  Fuzzing with Code Fragments , 2012, USENIX Security Symposium.

[2]  Alex Groce,et al.  Swarm testing , 2012, ISSTA 2012.

[3]  Xuejun Yang,et al.  Test-case reduction for C compiler bugs , 2012, PLDI.

[4]  Mariano Ceccato,et al.  An empirical study about the effectiveness of debugging when random test cases are used , 2012, 2012 34th International Conference on Software Engineering (ICSE).

[5]  Yang Xiang,et al.  Malware Variant Detection Using Similarity Search over Sets of Control Flow Graphs , 2011, 2011IEEE 10th International Conference on Trust, Security and Privacy in Computing and Communications.

[6]  Siau-Cheng Khoo,et al.  Towards more accurate retrieval of duplicate bug reports , 2011, 2011 26th IEEE/ACM International Conference on Automated Software Engineering (ASE 2011).

[7]  Xuejun Yang,et al.  Finding and understanding bugs in C compilers , 2011, PLDI '11.

[8]  Siau-Cheng Khoo,et al.  A discriminative model approach for accurate duplicate bug report retrieval , 2010, 2010 ACM/IEEE 32nd International Conference on Software Engineering.

[9]  Jacek Czerwonka,et al.  Test case comparison and clustering using program profiles and static execution , 2009, ESEC/FSE '09.

[10]  Weng-Keen Wong,et al.  Category detection using hierarchical mean shift , 2009, KDD.

[11]  Alex Groce,et al.  Random Test Run Length and Effectiveness , 2008, 2008 23rd IEEE/ACM International Conference on Automated Software Engineering.

[12]  Tao Xie,et al.  An approach to detecting duplicate bug reports using natural language and execution information , 2008, 2008 ACM/IEEE 30th International Conference on Software Engineering.

[13]  Mary Jean Harrold,et al.  Debugging in Parallel , 2007, ISSTA '07.

[14]  Nicholas Nethercote,et al.  Valgrind: a framework for heavyweight dynamic binary instrumentation , 2007, PLDI '07.

[15]  Alex Groce,et al.  Randomized Differential Testing as a Prelude to Formal Verification , 2007, 29th International Conference on Software Engineering (ICSE'07).

[16]  Yishay Mansour,et al.  Active sampling for multiple output identification , 2006, Machine Learning.

[17]  Chao Liu,et al.  Failure proximity: a fault localization-based approach , 2006, SIGSOFT '06/FSE-14.

[18]  Mohammad El-Ramly,et al.  Similarity in Programs , 2006, Duplication, Redundancy, and Similarity in Software.

[19]  Mary Jean Harrold,et al.  Empirical evaluation of the tarantula automatic fault-localization technique , 2005, ASE.

[20]  Kwangkeun Yi,et al.  Taming False Alarms from a Domain-Unaware C Analyzer by a Bayesian Statistical Post Analysis , 2005, SAS.

[21]  Michael I. Jordan,et al.  Scalable statistical bug isolation , 2005, PLDI '05.

[22]  H. Cleve,et al.  Locating causes of program failures , 2005, Proceedings. 27th International Conference on Software Engineering, 2005. ICSE 2005..

[23]  Alex Groce,et al.  Error explanation with distance metrics , 2004, International Journal on Software Tools for Technology Transfer.

[24]  Andrew W. Moore,et al.  Active Learning for Anomaly and Rare-Category Detection , 2004, NIPS.

[25]  David Leon,et al.  Tree-based methods for classifying software failures , 2004, 15th International Symposium on Software Reliability Engineering.

[26]  Alex Groce,et al.  Explaining abstract counterexamples , 2004, SIGSOFT '04/FSE-12.

[27]  Steven P. Reiss,et al.  Fault localization with nearest neighbor queries , 2003, 18th IEEE International Conference on Automated Software Engineering, 2003. Proceedings..

[28]  Dawson R. Engler,et al.  Z-Ranking: Using Statistical Analysis to Counter the Impact of Static Analysis Approximations , 2003, SAS.

[29]  Michael I. Jordan,et al.  Bug isolation via remote program sampling , 2003, PLDI '03.

[30]  Daniel Shawcross Wilkerson,et al.  Winnowing: local algorithms for document fingerprinting , 2003, SIGMOD '03.

[31]  Bin Wang,et al.  Automated support for classifying software failure reports , 2003, 25th International Conference on Software Engineering, 2003. Proceedings..

[32]  Joydeep Ghosh,et al.  Cluster Ensembles --- A Knowledge Reuse Framework for Combining Multiple Partitions , 2002, J. Mach. Learn. Res..

[33]  James A. Jones,et al.  Visualization of test information to assist fault localization , 2002, Proceedings of the 24th International Conference on Software Engineering. ICSE 2002.

[34]  Andreas Zeller,et al.  Simplifying and Isolating Failure-Inducing Input , 2002, IEEE Trans. Software Eng..

[35]  Andrew W. Moore,et al.  X-means: Extending K-means with Efficient Estimation of the Number of Clusters , 2000, ICML.

[36]  K. Claessen,et al.  QuickCheck: a lightweight tool for random testing of Haskell programs , 2000, ICFP '00.

[37]  W. M. McKeeman,et al.  Differential Testing for Software , 1998, Digit. Tech. J..

[38]  David B. Whalley,et al.  Automatic isolation of compiler errors , 1994, TOPL.

[39]  Teofilo F. GONZALEZ,et al.  Clustering to Minimize the Maximum Intercluster Distance , 1985, Theor. Comput. Sci..

[40]  Gerard Salton,et al.  A vector space model for automatic indexing , 1975, CACM.

[41]  Vladimir I. Levenshtein,et al.  Binary codes capable of correcting deletions, insertions, and reversals , 1965 .