Cooperative Bug Isolation

Debugging does not end with deployment. Static analysis, in-house testing, and good software engineering practices can catch or prevent many problems before software is distributed. Yet mainstream commercial software still ships with both known and unknown bugs. Real software still fails in the hands of real users. The need remains to identify and repair bugs that are only discovered, or whose importance is only revealed, after the software is released. Unfortunately we know almost nothing about how software behaves (and misbehaves) in the hands of end users. Traditional post-deployment feedback mechanisms, such as technical support phone calls or hand-composed bug reports, are informal, inconsistent, and highly dependent on manual, human intervention. This approach clouds the view, preventing engineers from seeing a complete and truly representative picture of how and why problems occur. This dissertation proposes a system to support debugging based on feedback from actual users. Cooperative Bug Isolation (CBI) leverages the key strength of user communities: their overwhelming numbers. We propose a low-overhead instrumentation strategy for gathering information from the executions experienced by large numbers of software end users. Our approach limits overhead using sparse random sampling rather than complete data collection, while simultaneously ensuring that the observed data is an unbiased, representative subset of the complete program behavior across all runs. We discuss a number of specific instrumentation schemes that may be coupled with the general sampling transformation to produce feedback data that we have found to be useful for isolating the causes of a wide variety of bugs. Collecting feedback from real code, especially real buggy code, is a nontrivial exercise. This dissertation presents our approach to a number of practical challenges that arise in building a complete, working CBI system. We discuss how the general sampling transformation scheme can be extended to deal with native compilers, libraries, dynamically loaded code, threads, and other features of modern software. We address questions of privacy and security as well as related issues of user interaction and informed user consent. This design and engineering investment has allowed us to begin an actual public deployment of a CBI system, initial results from which we report here. Of course, feedback data is only as useful as the sense we can make of it. When data is fair but very sparse, the noise level is high and traditional manual debugging techniques insufficient. This dissertation presents a suite of new algorithms for statistical debugging: finding and fixing software errors based on statistical analysis of sparse feedback data. The techniques vary in complexity and sophistication, from simple process of elimination strategies to regression techniques that build models of suspect program behaviors as failure predictors. Our most advanced technique combines a number of general and domain-specific statistical filtering and ranking techniques to separate the effects of different bugs and identify predictors that are associated with individual bugs. These predictors reveal both the circumstances under which bugs occur and the frequencies of failure modes, making it easier to prioritize debugging efforts. Our algorithm is validated using several case studies. These case studies include examples in which the algorithm found previously unknown, significant crashing bugs in widely used systems.

[1]  Leo Breiman,et al.  Classification and Regression Trees , 1984 .

[2]  Alexander Aiken,et al.  Program transformation in the presence of errors , 1989, POPL '90.

[3]  H. Hecht,et al.  Rare conditions and their effect on software failures , 1994, Proceedings of Annual Reliability and Maintainability Symposium (RAMS).

[4]  Eric Horvitz,et al.  Structure and chance: melding logic and probability for software debugging , 1995, CACM.

[5]  Martin C. Carlisle,et al.  Olden: parallelizing programs with dynamic data structures on distributed-memory machines , 1996 .

[6]  James R. Larus,et al.  Efficient path profiling , 1996, Proceedings of the 29th Annual IEEE/ACM International Symposium on Microarchitecture. MICRO 29.

[7]  Geoffrey Smith,et al.  A Type-Based Approach to Program Security , 1997, TAPSOFT.

[8]  H. Hecht,et al.  Toward more effective testing for high assurance systems , 1997, Proceedings 1997 High-Assurance Engineering Workshop.

[9]  Lance M. Berc,et al.  Continuous profiling: where have all the cycles gone? , 1997, TOCS.

[10]  William G. Griswold,et al.  Dynamically discovering likely program invariants to support program evolution , 1999, Proceedings of the 1999 International Conference on Software Engineering (IEEE Cat. No.99CB37002).

[11]  J. Mesirov,et al.  Molecular classification of cancer: class discovery and class prediction by gene expression monitoring. , 1999, Science.

[12]  Chrysanthos Dellarocas,et al.  Immunizing online reputation reporting systems against unfair ratings and discriminatory behavior , 2000, EC '00.

[13]  John Whaley,et al.  A portable sampling-based profiler for Java virtual machines , 2000, JAVA '00.

[14]  W. E. Weihl,et al.  Efficient and flexible value sampling , 2000, SIGP.

[15]  Matthew Arnold,et al.  A framework for reducing the cost of instrumented code , 2001, PLDI '01.

[16]  Trevor Hastie,et al.  The Elements of Statistical Learning , 2001 .

[17]  Andrew C. Myers,et al.  Untrusted hosts and confidentiality , 2001, SOSP.

[18]  Amitabh Srivastava,et al.  Vulcan Binary transformation in a distributed environment , 2001 .

[19]  Steven P. Reiss,et al.  Encoding program executions , 2001, Proceedings of the 23rd International Conference on Software Engineering. ICSE 2001.

[20]  Gerard J. Holzmann,et al.  Economics of software verification , 2001, PASTE '01.

[21]  Martin Hirzel,et al.  Bursty Tracing: A Framework for Low-Overhead Temporal Profiling , 2001 .

[22]  Alessandro Orso,et al.  Monitoring deployed software using software tomography , 2002, PASTE '02.

[23]  John F. Canny,et al.  Collaborative filtering with privacy , 2002, Proceedings 2002 IEEE Symposium on Security and Privacy.

[24]  J. Saxe,et al.  Extended static checking for Java , 2002, PLDI '02.

[25]  Sudheendra Hangal,et al.  Tracking down software bugs using automatic anomaly detection , 2002, ICSE '02.

[26]  Thomas A. Henzinger,et al.  Lazy abstraction , 2002, POPL '02.

[27]  R. Tibshirani,et al.  Diagnosis of multiple cancer types by shrunken centroids of gene expression , 2002, Proceedings of the National Academy of Sciences of the United States of America.

[28]  Peter M. Broadwell,et al.  Scrash: A System for Generating Secure Crash Information , 2003, USENIX Security Symposium.

[29]  Michael D. Ernst,et al.  Selecting Refining and Evaluating Properties for Program Analysis , 2003 .

[30]  Daniel Shawcross Wilkerson,et al.  Winnowing: local algorithms for document fingerprinting , 2003, SIGMOD '03.

[31]  Michael I. Jordan,et al.  Statistical Debugging of Sampled Programs , 2003, NIPS.

[32]  Shriram Krishnamurthi,et al.  Automated Fault Localization Using Potential Invariants , 2003, ArXiv.

[33]  Bin Wang,et al.  Automated support for classifying software failure reports , 2003, 25th International Conference on Software Engineering, 2003. Proceedings..

[34]  David A. Wagner,et al.  Finding User/Kernel Pointer Bugs with Type Inference , 2004, USENIX Security Symposium.

[35]  Ben Liblit,et al.  Public deployment of cooperative bug isolation , 2004, ICSE 2004.

[36]  Elaine J. Weyuker,et al.  Where the bugs are , 2004, ISSTA '04.

[37]  Static Checking of Dynamically Generated Queries in Database Applications , 2004 .

[38]  Matthias Hauswirth,et al.  Low-overhead memory leak detection using adaptive statistical profiling , 2004, ASPLOS XI.

[39]  Sebastian G. Elbaum,et al.  An empirical study of profiling strategies for released software and their impact on testing activities , 2004, ISSTA '04.

[40]  Gail C. Murphy,et al.  Predicting source code changes by mining change history , 2004, IEEE Transactions on Software Engineering.

[41]  Andreas Zeller,et al.  Mining Version Histories to Guide Software Changes , 2004 .