One λ at a Time: What Do We Know About Presenting Human-friendly Output from Program Analysis Tools?

Program analysis tools perform sophisticated analysis on source code to help programmers resolve compiler errors, apply optimizations, and identify security vulnerabilities. Despite the utility of these tools, research suggests that programmers do not frequently adopt them in practice—a primary reason being that the output of these tools is difficult to understand. Towards providing a synthesis of what researchers know about the presentation of program analysis output to programmers, we conducted a scoping review of the PLDI conference proceedings from 1988-2017. The scoping review serves as interim guidance for advancing collaborations between research disciplines. We discuss how cross-disciplinary communities, such as PLATEAU, are critical to improving the usability of program analysis tools.

[1]  Barbara Kitchenham,et al.  Procedures for Performing Systematic Reviews , 2004 .

[2]  Loris D'Antoni,et al.  Control-flow recovery from partial failure reports , 2017, PLDI.

[3]  Sumit Gulwani,et al.  Type-directed completion of partial expressions , 2012, PLDI.

[4]  Simon L. Peyton Jones,et al.  Diagnosing type errors with class , 2015, PLDI.

[5]  Steve Simmons,et al.  A new approach to debugging optimized code , 1992, PLDI '92.

[6]  Camil Demetrescu,et al.  Input-Sensitive Profiling , 2012, IEEE Transactions on Software Engineering.

[7]  Deborah S. Coutant,et al.  DOC: a practical approach to source-level debugging of globally optimized code , 1988, PLDI '88.

[8]  Robert W. Bowdidge,et al.  Programmers' build errors: a case study (at google) , 2014, ICSE.

[9]  Sam Blackshear,et al.  Almost-correct specifications: a modular semantic framework for assigning confidence to warnings , 2013, PLDI.

[10]  Xiaokang Qiu,et al.  Natural proofs for structure, data, and separation , 2013, PLDI.

[11]  David Van Horn,et al.  Relatively complete counterexamples for higher-order programs , 2015, PLDI.

[12]  Shan Lu,et al.  Automated atomicity-violation fixing , 2011, PLDI '11.

[13]  Susan Horwitz,et al.  Identifying the semantic and textual differences between two versions of a program , 1990, PLDI '90.

[14]  Sorin Lerner,et al.  Interactive parser synthesis by example , 2015, PLDI.

[15]  Anant Agarwal,et al.  TraceBack: first fault diagnosis by reconstruction of distributed control flow , 2005, PLDI '05.

[16]  Xiangyu Zhang,et al.  Pruning dynamic slices with confidence , 2006, PLDI '06.

[17]  Andrew C. Myers,et al.  Finding counterexamples from parsing conflicts , 2015, PLDI.

[18]  Daniel J. Dougherty,et al.  User Studies of Principled Model Finder Output , 2017, SEFM.

[19]  Amer Diwan,et al.  Explaining failures of program analyses , 2008, PLDI '08.

[20]  Patrick Th. Eugster,et al.  Semantics-aware trace analysis , 2009, PLDI '09.

[21]  Xuejun Yang,et al.  Test-case reduction for C compiler bugs , 2012, PLDI.

[22]  Isil Dillig,et al.  Automated error diagnosis using abductive inference , 2012, PLDI.

[23]  Emerson R. Murphy-Hill,et al.  Do Developers Read Compiler Error Messages? , 2017, 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE).

[24]  Maria J Grant,et al.  A typology of reviews: an analysis of 14 review types and associated methodologies. , 2009, Health information and libraries journal.

[25]  Santosh Nagarakatte,et al.  Precondition Inference for Peephole Optimizations in LLVM , 2016, ArXiv.

[26]  Shriram Krishnamurthi,et al.  Resugaring: lifting evaluation sequences through syntactic sugar , 2014, PLDI.

[27]  Thomas R. Gross,et al.  Source-level debugging of scalar optimized code , 1996, PLDI '96.

[28]  Manu Sridharan,et al.  TAJ: effective taint analysis of web applications , 2009, PLDI '09.

[29]  James R. Larus,et al.  Debugging temporal specifications with concept analysis , 2003, PLDI '03.

[30]  Alex Groce,et al.  Taming compiler fuzzers , 2013, PLDI.

[31]  Alan Edelman,et al.  PetaBricks: a language and compiler for algorithmic choice , 2009, PLDI '09.

[32]  Michael I. Jordan,et al.  Bug isolation via remote program sampling , 2003, PLDI.

[33]  Michael D. Bond,et al.  LeakChaser: helping programmers narrow down causes of memory leaks , 2011, PLDI '11.

[34]  Holger J Schünemann,et al.  Reviews: Rapid! Rapid! Rapid! …and systematic , 2015, Systematic Reviews.

[35]  Saturnino Garcia,et al.  Kremlin: rethinking and rebooting gprof for the multicore age , 2011, PLDI '11.

[36]  Sam Blackshear,et al.  Verification modulo versions: towards usable verification , 2014, PLDI.

[37]  Robert W. Bowdidge,et al.  Why don't software developers use static analysis tools to find bugs? , 2013, 2013 35th International Conference on Software Engineering (ICSE).

[38]  Olaf Chitil,et al.  Lightweight computation tree tracing for lazy functional languages , 2016, PLDI.

[39]  Michael I. Jordan,et al.  Scalable statistical bug isolation , 2005, PLDI '05.

[40]  Stefan Hanenberg,et al.  Faith, hope, and love: an essay on software science's neglect of human factors , 2010, OOPSLA.

[41]  Scott Moore,et al.  Exploring and enforcing security guarantees via program dependence graphs , 2015, PLDI.

[42]  Andrea C. Arpaci-Dusseau,et al.  Error propagation analysis for file systems , 2009, PLDI '09.

[43]  Emerson R. Murphy-Hill,et al.  Compiler error notifications revisited: an interaction-first approach for helping developers more effectively comprehend and resolve error notifications , 2014, ICSE Companion.

[44]  Kenneth L. McMillan,et al.  Ivy: safety verification by interactive generalization , 2016, PLDI.

[45]  Sebastian Burckhardt,et al.  It's alive! continuous feedback in UI programming , 2013, PLDI.

[46]  Benjamin Livshits,et al.  Merlin: specification inference for explicit information flow problems , 2009, PLDI '09.

[47]  Sigmund Cherem,et al.  Practical memory leak detection using guarded value-flow analysis , 2007, PLDI '07.

[48]  Matthias Felleisen,et al.  Catching bugs in the web of program invariants , 1996, PLDI '96.

[49]  V. Javier Traver,et al.  On Compiler Error Messages: What They Say and What They Mean , 2010, Adv. Hum. Comput. Interact..

[50]  Christian Bird,et al.  What developers want and need from program analysis: An empirical study , 2016, 2016 31st IEEE/ACM International Conference on Automated Software Engineering (ASE).

[51]  H. Arksey,et al.  Scoping studies: towards a methodological framework , 2005 .

[52]  Gerwin Klein,et al.  Don't sweat the small stuff: formal verification of C code without the pain , 2014, PLDI.

[53]  Eran Yahav,et al.  Chameleon: adaptive selection of collections , 2009, PLDI '09.

[54]  Rupak Majumdar,et al.  Cause clue clauses: error localization using maximum satisfiability , 2010, PLDI '11.

[55]  Satish Narayanasamy,et al.  Automatically classifying benign and harmful data races using replay analysis , 2007, PLDI '07.

[56]  Armando Solar-Lezama,et al.  Sketching concurrent data structures , 2008, PLDI '08.

[57]  Emina Torlak,et al.  MemSAT: checking axiomatic specifications of memory models , 2010, PLDI '10.

[58]  Peter J. Brown,et al.  Error messages: the neglected area of the man/machine interface , 1983, CACM.

[59]  Michael D. Bond,et al.  Breadcrumbs: efficient context sensitivity for dynamic bug detection analyses , 2010, PLDI '10.

[60]  Donald E. Porter,et al.  Improved error reporting for software that uses black-box components , 2007, PLDI '07.

[61]  Sumit Gulwani,et al.  Automated feedback generation for introductory programming assignments , 2012, PLDI.

[62]  Dan Grossman,et al.  Searching for type-error messages , 2007, PLDI '07.

[63]  Zhendong Su,et al.  Skeletal program enumeration for rigorous compiler testing , 2016, PLDI.