SherLock: unsupervised synchronization-operation inference

Synchronizations are fundamental to the correctness and performance of concurrent software. They determine which operations can execute concurrently and which can-not—the key to detecting and fixing concurrency bugs, as well as understanding and tuning performance. Unfortunately, correctly identifying all synchronizations has become extremely difficult in modern software systems due to the various forms of concurrency and various types of synchronizations. Previous work either only infers specific type of synchronization by code analysis or relies on manual effect to annotate the synchronization. This paper proposes SherLock, a tool that automatically infers synchronizations without code analysis or annotation. SherLock leverages the fact that most synchronizations appear around the conflicting operations and encodes the inference problem into a linear system with properties and hypotheses about how synchronizations are typically used. To collect useful observations, SherLock runs the target problem for a small number of runs with feedback-guided delay injection. We have applied SherLock on 8 C# open-source applications. Without any prior knowledge, SherLock automatically inferred more than 120 unique synchronizations, with few false positives. These inferred synchronizations cover a wide variety of types, including lock operations, fork-join operations, asynchronous operations, framework synchronization, and custom synchronization.

[1]  Guangming Zeng,et al.  SyncPerf: Categorizing, Detecting, and Diagnosing Synchronization Performance Bugs , 2017, EuroSys.

[2]  Michael I. Jordan,et al.  Scalable statistical bug isolation , 2005, PLDI '05.

[3]  Julia L. Lawall,et al.  Continuously measuring critical section pressure with the free-lunch profiler , 2014, OOPSLA.

[4]  Pravesh Kothari,et al.  A randomized scheduler with probabilistic guarantees of finding bugs , 2010, ASPLOS XV.

[5]  Satish Narayanasamy,et al.  Race detection for event-driven mobile applications , 2014, PLDI.

[6]  M. Hill,et al.  Weak ordering-a new definition , 1990, [1990] Proceedings. The 17th Annual International Symposium on Computer Architecture.

[7]  Accelerating critical section execution with asymmetric multi-core architectures , 2009, ASPLOS.

[8]  Leslie Lamport,et al.  Time, clocks, and the ordering of events in a distributed system , 1978, CACM.

[9]  Benjamin Livshits,et al.  Merlin: specification inference for explicit information flow problems , 2009, PLDI '09.

[10]  Martin T. Vechev,et al.  Scalable taint specification inference with big code , 2019, PLDI.

[11]  Satish Narayanasamy,et al.  AsyncClock: Scalable Inference of Asynchronous Event Causality , 2017, ASPLOS.

[12]  C. Flanagan The FASTTRACK2 Race Detector , 2017 .

[13]  Xu Chen,et al.  Automating Network Application Dependency Discovery: Experiences, Limitations, and New Solutions , 2008, OSDI.

[14]  Michael Burrows,et al.  Eraser: a dynamic data race detector for multi-threaded programs , 1997, TOCS.

[15]  Grigore Rosu,et al.  jPredictor , 2008, 2008 ACM/IEEE 30th International Conference on Software Engineering.

[16]  Rajiv Gupta,et al.  Dynamic recognition of synchronization operations for improved data race detection , 2008, ISSTA '08.

[17]  Koen De Bosschere,et al.  RecPlay: a fully integrated practical record/replay system , 1999, TOCS.

[18]  Wei Zhang,et al.  Automated Concurrency-Bug Fixing , 2012, OSDI.

[19]  Eric Bodden,et al.  A Machine-learning Approach for Classifying and Categorizing Android Sources and Sinks , 2014, NDSS.

[20]  Sudheendra Hangal,et al.  Tracking down software bugs using automatic anomaly detection , 2002, ICSE '02.

[21]  Shan Lu,et al.  ConSeq: detecting concurrency bugs through sequential errors , 2011, ASPLOS XVI.

[22]  Suman Nath,et al.  Efficient Scalable Thread-Safety-Violation Detection , 2019 .

[23]  George Candea,et al.  Data races vs. data race bugs: telling the difference with portend , 2012, ASPLOS XVII.

[24]  Yuanyuan Zhou,et al.  AVIO: Detecting Atomicity Violations via Access-Interleaving Invariants , 2007, IEEE Micro.

[25]  Manu Sridharan,et al.  Race detection for web applications , 2012, PLDI.

[26]  Satish Narayanasamy,et al.  DoublePlay: parallelizing sequential logging and replay , 2011, ASPLOS XVI.

[27]  William G. Griswold,et al.  Quickly detecting relevant program invariants , 2000, Proceedings of the 2000 International Conference on Software Engineering. ICSE 2000 the New Millennium.

[28]  Dan Grossman,et al.  Low-level detection of language-level data races with LARD , 2014, ASPLOS.

[29]  Dawson R. Engler,et al.  Bugs as deviant behavior: a general approach to inferring errors in systems code , 2001, SOSP.

[30]  Shan Lu,et al.  DFix: automatically fixing timing bugs in distributed systems , 2019, PLDI.

[31]  Zhiqiang Ma,et al.  Ad Hoc Synchronization Considered Harmful , 2010, OSDI.

[32]  Brandon Lucia,et al.  Atom-Aid: Detecting and Surviving Atomicity Violations , 2009, IEEE Micro.

[33]  Shan Lu,et al.  Automated atomicity-violation fixing , 2011, PLDI '11.

[34]  Rupak Majumdar,et al.  Race detection for Android applications , 2014, PLDI.

[35]  Horatiu Jula,et al.  Deadlock Immunity: Enabling Systems to Defend Against Deadlocks , 2008, OSDI.

[36]  Stephen N. Freund,et al.  FastTrack: efficient and precise dynamic race detection , 2009, PLDI '09.

[37]  Thomas F. Wenisch,et al.  The Mystery Machine: End-to-end Performance Analysis of Large-scale Internet Services , 2014, OSDI.

[38]  Yang Liu,et al.  ATDetector: Improving the accuracy of a commercial data race detector by identifying address transfer , 2011, 2011 44th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO).

[39]  Shan Lu,et al.  Pcatch: automatically detecting performance cascading bugs in cloud systems , 2018, EuroSys.