GoBench: A Benchmark Suite of Real-World Go Concurrency Bugs

Go, a fast growing programming language, is often considered as “the programming language of the cloud”. The language provides a rich set of synchronization primitives, making it easy to write concurrent programs with great parallelism. However. the rich set of primitives also introduces many bugs. We build Gobench, the first benchmark suite for Go concurrency bugs. Currently, Gobench consists of 82 real bugs from 9 popular open source applications and 103 bug kernels. The bug kernels are carefully extracted and simplified from 67 out of these 82 bugs and 36 additional bugs reported in a recent study to preserve their bug-inducing complexities as much as possible. These bugs cover a variety of concurrency issues, both traditional and Go-specific. We believe Gobench will be instrumental in helping researchers understand concurrency bugs in Go and develop effective tools for their detection. We have therefore evaluated a range of representative concurrency error detection tools using Gobench. Our evaluation has revealed their limitations and provided insights for making further improvements.

[1]  Shan Lu,et al.  DCatch: Automatically Detecting Distributed Concurrency Bugs in Cloud Systems , 2017, ASPLOS.

[2]  Peter Thiemann,et al.  Static Trace-Based Deadlock Analysis for Synchronous Mini-Go , 2016, APLAS.

[3]  Shing-Tsaan Huang,et al.  A distributed deadlock detection algorithm for CSP-like communication , 1990, TOPL.

[4]  Flemming Nielson,et al.  Process-Local Static Analysis of Synchronous Processes , 2018, SAS.

[5]  Koushik Sen,et al.  A randomized dynamic program analysis technique for detecting real deadlocks , 2009, PLDI '09.

[6]  Arie Shoshani,et al.  System Deadlocks , 1971, CSUR.

[7]  David Lie,et al.  Kivati: fast detection and prevention of atomicity violations , 2010, EuroSys '10.

[8]  Yuanyuan Zhou,et al.  AVIO: Detecting Atomicity Violations via Access-Interleaving Invariants , 2007, IEEE Micro.

[9]  Shan Lu,et al.  TaxDC: A Taxonomy of Non-Deterministic Concurrency Bugs in Datacenter Distributed Systems , 2016, ASPLOS.

[10]  C. A. R. Hoare,et al.  Communicating sequential processes , 1978, CACM.

[11]  Leslie Lamport,et al.  Time, clocks, and the ordering of events in a distributed system , 1978, CACM.

[12]  Bernardo Toninho,et al.  Fencing off go: liveness and safety for channel-based programming , 2016, POPL.

[13]  Mukesh Singhal,et al.  Deadlock detection in distributed systems , 1989, Computer.

[14]  Robbert Krebbers,et al.  Actris: session-type based reasoning in separation logic , 2019, Proc. ACM Program. Lang..

[15]  Jianjun Zhao,et al.  JaConTeBe: A Benchmark Suite of Real-World Java Concurrency Bugs (T) , 2015, 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE).

[16]  Nicolas Dilley,et al.  An Empirical Study of Messaging Passing Concurrency in Go Projects , 2019, 2019 IEEE 26th International Conference on Software Analysis, Evolution and Reengineering (SANER).

[17]  Ian Karlin,et al.  Exploring Regression of Data Race Detection Tools Using DataRaceBench , 2019, 2019 IEEE/ACM 3rd International Workshop on Software Correctness for HPC Applications (Correctness).

[18]  Konstantin Serebryany,et al.  ThreadSanitizer: data race detection in practice , 2009, WBIA '09.

[19]  Xiaobing Feng,et al.  CrashTuner: detecting crash-recovery bugs in cloud systems via meta-info analysis , 2019, SOSP.

[20]  Murali Krishna Ramanathan,et al.  Trace driven dynamic deadlock detection and reproduction , 2014, PPoPP.

[21]  Bernardo Toninho,et al.  A Static Verification Framework for Message Passing in Go Using Behavioural Types , 2018, 2018 IEEE/ACM 40th International Conference on Software Engineering (ICSE).

[22]  Daniel Brand,et al.  On Communicating Finite-State Machines , 1983, JACM.

[23]  Qing Zhou,et al.  May-happen-in-parallel analysis with static vector clocks , 2018, CGO.

[24]  Nobuko Yoshida,et al.  Static deadlock detection for concurrent go by global session graph synthesis , 2016, CC.

[25]  Alexey Lastovetsky.  The mpC Programming Language Specification , 1994 .

[26]  Daniel Kroening,et al.  Sound static deadlock analysis for C/Pthreads , 2016, 2016 31st IEEE/ACM International Conference on Automated Software Engineering (ASE).

[27]  David Gay,et al.  Effective static deadlock detection , 2009, 2009 IEEE 31st International Conference on Software Engineering.

[28]  Yiying Zhang,et al.  Understanding Real-World Concurrency Bugs in Go , 2019, ASPLOS.

[29]  Sebastian Burckhardt,et al.  Effective Data-Race Detection for the Kernel , 2010, OSDI.

[30]  Xian Zhang,et al.  JBench: A Dataset of Data Races for Concurrency Testing , 2018, 2018 IEEE/ACM 15th International Conference on Mining Software Repositories (MSR).

[31]  David Gay,et al.  An effective dynamic analysis for detecting generalized deadlocks , 2010, FSE '10.

[32]  Yannis Smaragdakis,et al.  Sound predictive race detection in polynomial time , 2012, POPL '12.

[33]  Shan Lu,et al.  FlyMC: Highly Scalable Testing of Complex Interleavings in Distributed Systems , 2019, EuroSys.

[34]  Yan Cai,et al.  Dynamic Testing for Deadlocks via Constraints , 2016, IEEE Transactions on Software Engineering.

[35]  Tanakorn Leesatapornwongsa,et al.  SAMC: a fast model checker for finding heisenbugs in distributed systems (demo) , 2015, ISSTA.

[36]  Koushik Sen,et al.  RADBench: A Concurrency Bug Benchmark Suite , 2011, HotPar.

[37]  Feng Li,et al.  CloudRaid: hunting concurrency bugs in the cloud via log-mining , 2018, ESEC/SIGSOFT FSE.

[38]  Jingling Xue,et al.  Acculock: Accurate and efficient detection of data races , 2011, International Symposium on Code Generation and Optimization (CGO 2011).

[39]  Yi Deng,et al.  Optimal deadlock detection in distributed systems based on locally constructed wait-for graphs , 1996, Proceedings of 16th International Conference on Distributed Computing Systems.