MUX: algorithm selection for software model checkers

With the growing complexity of modern day software, software model checking has become a critical technology for ensuring correctness of software. As is true with any promising technology, there are a number of tools for software model checking. However, their respective performance trade-offs are difficult to characterize accurately – making it difficult for practitioners to select a suitable tool for the task at hand. This paper proposes a technique called MUX that addresses the problem of selecting the most suitable software model checker for a given input instance. MUX performs machine learning on a repository of software verification instances. The algorithm selector, synthesized through machine learning, uses structural features from an input instance, comprising a program-property pair, at runtime and determines which tool to use. We have implemented MUX for Windows device drivers and evaluated it on a number of drivers and model checkers. Our results are promising in that the algorithm selector not only avoids a significant number of timeouts but also improves the total runtime by a large margin, compared to any individual model checker. It also outperforms a portfolio-based algorithm selector being used in Microsoft at present. Besides, MUX identifies structural features of programs that are key factors in determining performance of model checkers.

[1]  Aditya V. Thakur,et al.  The Yogi Project : Software Property Checking via Static Analysis and Testing , 2009 .

[2]  Tad Hogg,et al.  An Economics Approach to Hard Computational Problems , 1997, Science.

[3]  Albert L. Baker,et al.  Preliminary design of JML: a behavioral interface specification language for java , 2006, SOEN.

[4]  Y. N. Srikant,et al.  Microarchitecture Sensitive Empirical Models for Compiler Optimizations , 2007, International Symposium on Code Generation and Optimization (CGO'07).

[5]  Thomas A. Henzinger,et al.  Software Verification with BLAST , 2003, SPIN.

[6]  Ian H. Witten,et al.  The WEKA data mining software: an update , 2009, SKDD.

[7]  Dirk Beyer,et al.  Second Competition on Software Verification - (Summary of SV-COMP 2013) , 2013, TACAS.

[8]  Sriram K. Rajamani,et al.  SLAM and Static Driver Verifier: Technology Transfer of Formal Methods inside Microsoft , 2004, IFM.

[9]  John R. Rice,et al.  The Algorithm Selection Problem , 1976, Adv. Comput..

[10]  Kevin Leyton-Brown,et al.  SATzilla: Portfolio-based Algorithm Selection for SAT , 2008, J. Artif. Intell. Res..

[11]  Klaus Havelund,et al.  Model checking programs , 2000, Proceedings ASE 2000. Fifteenth IEEE International Conference on Automated Software Engineering.

[12]  Marat Boshernitsan,et al.  Predicting Effectiveness of Automatic Testing Tools , 2008, 2008 23rd IEEE/ACM International Conference on Automated Software Engineering.

[13]  W. Hsu,et al.  Algorithm selection for sorting and probabilistic inference: a machine learning-based approach , 2003 .

[14]  Christopher M. Bishop,et al.  Pattern Recognition and Machine Learning (Information Science and Statistics) , 2006 .

[15]  Bernd Bischl,et al.  Algorithm selection based on exploratory landscape analysis and cost-sensitive learning , 2012, GECCO '12.

[16]  Nasser M. Nasrabadi,et al.  Pattern Recognition and Machine Learning , 2006, Technometrics.

[17]  Marsha Chechik,et al.  UFO: Verification with Interpolants and Abstract Interpretation - (Competition Contribution) , 2013, TACAS.

[18]  Tim Menzies,et al.  Data Mining Static Code Attributes to Learn Defect Predictors , 2007, IEEE Transactions on Software Engineering.

[19]  Chih-Jen Lin,et al.  LIBLINEAR: A Library for Large Linear Classification , 2008, J. Mach. Learn. Res..

[20]  David Lo,et al.  Will Fault Localization Work for These Failures? An Automated Approach to Predict Effectiveness of Fault Localization Tools , 2013, 2013 IEEE International Conference on Software Maintenance.

[21]  Michela Milano,et al.  Learning Techniques for Automatic Algorithm Portfolio Selection , 2004, ECAI.

[22]  Shuvendu K. Lahiri,et al.  A Solver for Reachability Modulo Theories , 2012, CAV.

[23]  Gianpiero Cabodi,et al.  Thread-based multi-engine model checking for multicore platforms , 2013, TODE.

[24]  Thomas A. Henzinger,et al.  Conditional model checking: a technique to pass information between verifiers , 2012, SIGSOFT FSE.

[25]  Nancy M. Amato,et al.  A framework for adaptive algorithm selection in STAPL , 2005, PPoPP.

[26]  Kate Smith-Miles,et al.  Towards insightful algorithm selection for optimisation using meta-learning concepts , 2008, 2008 IEEE International Joint Conference on Neural Networks (IEEE World Congress on Computational Intelligence).

[27]  Brahim Hnich,et al.  Making Choices Using Structure at the Instance Level within a Case Based Reasoning Framework , 2004, CPAIOR.

[28]  Horst Samulowitz,et al.  Learning to Solve QBF , 2007, AAAI.

[29]  M. Kendall,et al.  Rank Correlation Methods , 1949 .

[30]  C. F. Kossack,et al.  Rank Correlation Methods , 1949 .

[31]  Rudolf Eigenmann,et al.  Context-sensitive domain-independent algorithm composition and selection , 2006, PLDI '06.

[32]  Patrice Godefroid,et al.  Software Model Checking: The VeriSoft Approach , 2005, Formal Methods Syst. Des..

[33]  Jia Feng-jun.  Specification and Description Language(SDL) and Its Applications to Development of TD-PRS Mobile Terminals , 2003 .

[34]  Chih-Jen Lin,et al.  LIBSVM: A library for support vector machines , 2011, TIST.

[35]  Michail G. Lagoudakis,et al.  Algorithm Selection using Reinforcement Learning , 2000, ICML.

[36]  Yuri Malitsky,et al.  Algorithm Selection and Scheduling , 2011, CP.

[37]  Sriram K. Rajamani,et al.  SLIC: A Specification Language for Interface Checking (of C) , 2002 .

[38]  Dawson R. Engler,et al.  Proceedings of the 5th Symposium on Operating Systems Design and Implementation Cmc: a Pragmatic Approach to Model Checking Real Code , 2022 .

[39]  David Lo,et al.  Search-based fault localization , 2011, 2011 26th IEEE/ACM International Conference on Automated Software Engineering (ASE 2011).

[40]  Thomas A. Henzinger,et al.  Configurable Software Verification: Concretizing the Convergence of Model Checking and Program Analysis , 2007, CAV.

[41]  S. Rajamani,et al.  A decade of software model checking with SLAM , 2011, Commun. ACM.

[42]  A. Turing.  On Computable Numbers, with an Application to the Entscheidungsproblem. , 1937 .

[43]  Sriram K. Rajamani,et al.  An empirical study of optimizations in YOGI , 2010, 2010 ACM/IEEE 32nd International Conference on Software Engineering.

[44]  Douglas L. Jones,et al.  Fast searches for effective optimization phase sequences , 2004, PLDI '04.

[45]  Sven Apel,et al.  Domain Types: Abstract-Domain Selection Based on Variable Usage , 2013, Haifa Verification Conference.

[46]  Itu-T.  Specification and Description Language (SDL) , 1999 .

[47]  Thomas Ball,et al.  SLAM2: Static driver verification with under 4% false alarms , 2010, Formal Methods in Computer Aided Design.