Improving Software Diagnosability via Log Enhancement

Diagnosing software failures in the field is notoriously difficult, in part due to the fundamental complexity of trouble-shooting any complex software system, but further exacerbated by the paucity of information that is typically available in the production setting. Indeed, for reasons of both overhead and privacy, it is common that only the run-time log generated by a system (e.g., syslog) can be shared with the developers. Unfortunately, the ad-hoc nature of such reports are frequently insufficient for detailed failure diagnosis. This paper seeks to improve this situation within the rubric of existing practice. We describe a tool, LogEnhancer that automatically "enhances" existing logging code to aid in future post-failure debugging. We evaluate LogEnhancer on eight large, real-world applications and demonstrate that it can dramatically reduce the set of potential root failure causes that must be considered during diagnosis while imposing negligible overheads.

[1]  Alfred V. Aho,et al.  Compilers: Principles, Techniques, and Tools , 1986, Addison-Wesley series in computer science / World student series edition.

[2]  Thomas J. LeBlanc,et al.  Debugging Parallel Programs with Instant Replay , 1987, IEEE Transactions on Computers.

[3]  K. Rustan M. Leino,et al.  Extended static checking , 1998, PROCOMET.

[4]  Brian W. Kernighan,et al.  The Practice of Programming , 1999 .

[5]  Greg Nelson,et al.  Extended static checking for Java , 2002, PLDI '02.

[6]  George C. Necula,et al.  CIL: Intermediate Language and Tools for Analysis and Transformation of C Programs , 2002, CC.

[7]  Marcos K. Aguilera,et al.  Performance debugging for distributed systems of black boxes , 2003, SOSP '03.

[8]  Michael I. Jordan,et al.  Bug isolation via remote program sampling , 2003, PLDI '03.

[9]  Dawson R. Engler,et al.  RacerX: effective, static detection of race conditions and deadlocks , 2003, SOSP '03.

[10]  Min Xu,et al.  A "flight data recorder" for enabling full-system multiprocessor deterministic replay , 2003, ISCA '03.

[11]  Manu Sridharan,et al.  PSE: explaining program failures via postmortem static analysis , 2004, SIGSOFT '04/FSE-12.

[12]  Richard Mortier,et al.  Using Magpie for Request Extraction and Workload Modelling , 2004, OSDI.

[13]  Satish Narayanasamy,et al.  BugNet: continuously recording program execution for deterministic replay debugging , 2005, 32nd International Symposium on Computer Architecture (ISCA'05).

[14]  Anant Agarwal,et al.  TraceBack: first fault diagnosis by reconstruction of distributed control flow , 2005, PLDI '05.

[15]  Armando Fox,et al.  Capturing, indexing, clustering, and retrieving system history , 2005, SOSP '05.

[16]  Samuel T. King,et al.  Debugging Operating Systems with Time-Traveling Virtual Machines (Awarded General Track Best Paper Award!) , 2005, USENIX Annual Technical Conference, General Track.

[17]  Xiangyu Zhang,et al.  Dynamic slicing long running programs through execution fast forwarding , 2006, SIGSOFT '06/FSE-14.

[18]  Alexander Aiken,et al.  How is aliasing used in systems software? , 2006, SIGSOFT '06/FSE-14.

[19]  Yuanyuan Zhou,et al.  Have things changed now?: an empirical study of bug characteristics in modern open source software , 2006, ASID '06.

[20]  Emery D. Berger,et al.  DieHard: probabilistic memory safety for unsafe languages , 2006, PLDI '06.

[21]  Alexander Aiken,et al.  Effective static race detection for Java , 2006, PLDI '06.

[22]  Alfred V. Aho,et al.  Compilers: Principles, Techniques, and Tools (2nd Edition) , 2006 .

[23]  Isil Dillig,et al.  An overview of the saturn project , 2007, PASTE '07.

[24]  Alexander Aiken,et al.  Conditional must not aliasing for static race detection , 2007, ACM-SIGACT Symposium on Principles of Programming Languages.

[25]  Alexander Aiken,et al.  Conditional must not aliasing for static race detection , 2007, POPL '07.

[26]  Donald E. Porter,et al.  Improved error reporting for software that uses black-box components , 2007, PLDI '07.

[27]  Ozalp Babaoglu,et al.  ACM Transactions on Computer Systems , 2007 .

[28]  Yuanyuan Zhou,et al.  Triage: diagnosing production run failures at the user's site , 2007, SOSP.

[29]  Emery D. Berger,et al.  Exterminator: automatically correcting memory errors with high probability , 2007, PLDI '07.

[30]  Ling Huang,et al.  Mining Console Logs for Large-Scale System Problem Detection , 2008, SysML.

[31]  Josep Torrellas,et al.  DeLorean: Recording and Deterministically Replaying Shared-Memory Multiprocessor Execution Ef?ciently , 2008, 2008 International Symposium on Computer Architecture.

[32]  Xuezheng Liu,et al.  Usenix Association 8th Usenix Symposium on Operating Systems Design and Implementation R2: an Application-level Kernel for Record and Replay , 2022 .

[33]  Miguel Castro,et al.  Better bug reporting with better privacy , 2008, ASPLOS 2008.

[34]  Abhishek Kumar,et al.  Lightweight, High-Resolution Monitoring for Troubleshooting Production Systems , 2008, OSDI.

[35]  Larry Rudolph,et al.  How to Do a Million Watchpoints: Efficient Debugging Using Dynamic Instrumentation , 2008, CC.

[36]  Josep Torrellas,et al.  DeLorean: Recording and Deterministically Replaying Shared-Memory Multiprocessor Execution Ef?ciently , 2008, International Symposium on Computer Architecture.

[37]  Manuel Costa,et al.  Bouncer: securing software by blocking bad input , 2008, WRAITS '08.

[38]  Dawson R. Engler,et al.  KLEE: Unassisted and Automatic Generation of High-Coverage Tests for Complex Systems Programs , 2008, OSDI.

[39]  Peter M. Chen,et al.  Execution replay of multiprocessor virtual machines , 2008, VEE '08.

[40]  Babak Falsafi,et al.  Flexible Hardware Acceleration for Instruction-Grain Program Monitoring , 2008, 2008 International Symposium on Computer Architecture.

[41]  Galen C. Hunt,et al.  Debugging in the (very) large: ten years of implementation and experience , 2009, SOSP '09.

[42]  Asim Kadav,et al.  Tolerating hardware device failures in software , 2009, SOSP '09.

[43]  Marek Olszewski,et al.  Kendo: efficient deterministic multithreading in software , 2009, ASPLOS.

[44]  Yuanyuan Zhou,et al.  PRES: probabilistic replay with execution sketching on multiprocessors , 2009, SOSP '09.

[45]  Trishul M. Chilimbi,et al.  HOLMES: Effective statistical debugging via efficient path profiling , 2009, 2009 IEEE 31st International Conference on Software Engineering.

[46]  Michael I. Jordan,et al.  Detecting large-scale system problems by mining console logs , 2009, SOSP '09.

[47]  Satish Narayanasamy,et al.  Respec: efficient online multiprocessor replayvia speculation and external determinism , 2010, ASPLOS XV.

[48]  George Candea,et al.  Execution synthesis: a technique for automated software debugging , 2010, EuroSys '10.

[49]  Brandon Lucia,et al.  DMP: Deterministic Shared-Memory Multiprocessing , 2010, IEEE Micro.

[50]  David Lie,et al.  Kivati: fast detection and prevention of atomicity violations , 2010, EuroSys '10.

[51]  Babak Falsafi,et al.  ParaLog: enabling and accelerating online parallel monitoring of multithreaded applications , 2010, ASPLOS XV.

[52]  George Candea,et al.  Low-Overhead Bug Fingerprinting for Fast Debugging , 2010, RV.

[53]  Satish Narayanasamy,et al.  Respec: Efficient Online Multiprocessor Replay via Speculation and External Determinism , 2010, ASPLOS 2010.

[54]  Ding Yuan,et al.  SherLog: error diagnosis by connecting clues from run-time logs , 2010, ASPLOS XV.

[55]  Brandon Lucia,et al.  ColorSafe: architectural support for debugging and dynamically avoiding multi-variable atomicity violations , 2010, ISCA.

[56]  Xiangyu Zhang,et al.  Analyzing multicore dumps to facilitate concurrency bug reproduction , 2010, ASPLOS XV.

[57]  Jason Nieh,et al.  Record and transplay: partial checkpointing for replay debugging across heterogeneous systems , 2011, SIGMETRICS '11.

[58]  Satish Narayanasamy,et al.  DoublePlay: parallelizing sequential logging and replay , 2011, ASPLOS XVI.

[59]  Ricardo Bianchini,et al.  Striking a new balance between program instrumentation and debugging time , 2011, EuroSys '11.

[60]  Jason Nieh,et al.  Record and transplay: partial checkpointing for replay debugging across heterogeneous systems , 2011, PERV.