Fault Localization for Declarative Models in Alloy

Fault localization is a popular research topic and many techniques have been proposed to locate faults in imperative code, e.g. C and Java. In this paper, we focus on the problem of fault localization for declarative models in Alloy – a first order relational logic with transitive closure. We introduce AlloyFLhy, the first fault localization technique for faulty Alloy models which leverages multiple test formulas. AlloyFLhy brings the traditional spectrum-based and mutation-based fault localization techniques to Alloy and combines both techniques to locate faults. To measure the effectiveness of AlloyFLhy, we define three distance metrics and use both distance-based and top-k metrics to measure the effectiveness of AlloyFLhy on 90 real faulty models. The results show that AlloyFLhy is substantially more effective than Alloy’s built-in unsat core.

[1]  Alessandro Orso,et al.  Are automated debugging techniques actually helping programmers? , 2011, ISSTA '11.

[2]  Wes Masri,et al.  Fault localization based on information flow coverage , 2010, Softw. Test. Verification Reliab..

[3]  W. Eric Wong,et al.  The DStar Method for Effective Software Fault Localization , 2014, IEEE Transactions on Reliability.

[4]  Chao Liu,et al.  SOBER: statistical model-based bug localization , 2005, ESEC/FSE-13.

[5]  Natali Ruchansky,et al.  A (not) NICE way to verify the openflow switch specification: formal modelling of the openflow switch using alloy , 2013, SIGCOMM.

[6]  Shin Yoo,et al.  Ask the Mutants: Mutating Faulty Programs for Fault Localization , 2014, 2014 IEEE Seventh International Conference on Software Testing, Verification and Validation.

[7]  Felix Sheng-Ho Chang,et al.  Finding Minimal Unsatisfiable Cores of Declarative Specifications , 2008, FM.

[8]  Bernhard Rumpe,et al.  CD2Alloy: class diagrams analysis using alloy revisited , 2011, MODELS'11.

[9]  Sarfraz Khurshid,et al.  MuAlloy: A Mutation Testing Framework for Alloy , 2018, 2018 IEEE/ACM 40th International Conference on Software Engineering: Companion (ICSE-Companion).

[10]  Mark Harman,et al.  Higher Order Mutation Testing , 2009, Inf. Softw. Technol..

[11]  Sarfraz Khurshid,et al.  Automated Model Repair for Alloy , 2018, 2018 33rd IEEE/ACM International Conference on Automated Software Engineering (ASE).

[12]  Yves Le Traon,et al.  Metallaxis‐FL: mutation‐based fault localization , 2015, Softw. Test. Verification Reliab..

[13]  Kathi Fisler,et al.  The Margrave Tool for Firewall Analysis , 2010, LISA.

[14]  Sarfraz Khurshid,et al.  Improving the effectiveness of spectra-based fault localization using specifications , 2012, 2012 Proceedings of the 27th IEEE/ACM International Conference on Automated Software Engineering.

[15]  Derek Rayside,et al.  Staged Evaluation of Partial Instances in a Relational Model Finder , 2014, ABZ.

[16]  Sarfraz Khurshid,et al.  Towards a test automation framework for alloy , 2014, SPIN.

[17]  Dianxiang Xu,et al.  Towards Better Fault Localization: A Crosstab-Based Statistical Approach , 2012, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[18]  Albert L. Baker,et al.  JML: A Notation for Detailed Design , 1999, Behavioral Specifications of Businesses and Systems.

[19]  Emina Torlak,et al.  Kodkod: A Relational Model Finder , 2007, TACAS.

[20]  Martin Monperrus,et al.  Learning to Combine Multiple Ranking Metrics for Fault Localization , 2014, 2014 IEEE International Conference on Software Maintenance and Evolution.

[21]  David Lo,et al.  Should I follow this fault localization tool’s output? , 2014, Empirical Software Engineering.

[22]  David Lo,et al.  Theory and Practice, Do They Match? A Case with Spectrum-Based Fault Localization , 2013, 2013 IEEE International Conference on Software Maintenance.

[23]  Manu Sridharan,et al.  Debugging overconstrained declarative models using unsatisfiable cores , 2003, 18th IEEE International Conference on Automated Software Engineering, 2003. Proceedings..

[24]  Rajiv Gupta,et al.  Fault localization using value replacement , 2008, ISSTA '08.

[25]  Raúl A. Santelices,et al.  Lightweight fault-localization using multiple coverage types , 2009, 2009 IEEE 31st International Conference on Software Engineering.

[26]  Xiangyu Zhang,et al.  Locating faults through automated predicate switching , 2006, ICSE.

[27]  Lars Grunske,et al.  A Critical Evaluation of Spectrum-Based Fault Localization Techniques on a Large-Scale Software System , 2017, 2017 IEEE International Conference on Software Quality, Reliability and Security (QRS).

[28]  Peter Zoeteweij,et al.  A practical evaluation of spectrum-based fault localization , 2009, J. Syst. Softw..

[29]  Sarfraz Khurshid,et al.  AUnit: A Test Automation Tool for Alloy , 2018, 2018 IEEE 11th International Conference on Software Testing, Verification and Validation (ICST).

[30]  Mary Jean Harrold,et al.  Empirical evaluation of the tarantula automatic fault-localization technique , 2005, ASE.

[31]  Sarfraz Khurshid,et al.  Automated Test Generation and Mutation Testing for Alloy , 2017, 2017 IEEE International Conference on Software Testing, Verification and Validation (ICST).

[32]  Spencer Pearson,et al.  Evaluation of fault localization techniques , 2016, SIGSOFT FSE.

[33]  Gade Krishna,et al.  A scalable peer-to-peer lookup protocol for Internet applications , 2012 .

[34]  Xiangyu Zhang,et al.  Locating faulty code using failure-inducing chops , 2005, ASE.

[35]  A. Zeller.  Isolating cause-effect chains from computer programs , 2002, SIGSOFT '02/FSE-10.

[36]  Michael D. Ernst,et al.  An Empirical Study of Fault Localization Families and Their Combinations , 2018, IEEE Transactions on Software Engineering.

[37]  John T. Stasko,et al.  Visualization of test information to assist fault localization , 2002, ICSE '02.

[38]  Sarfraz Khurshid,et al.  Systematic Generation of Non-equivalent Expressions for Relational Algebra , 2018, ABZ.

[39]  Peter Zoeteweij,et al.  Spectrum-Based Multiple Fault Localization , 2009, 2009 IEEE/ACM International Conference on Automated Software Engineering.

[40]  Jun Sun,et al.  Feedback-Based Debugging , 2017, 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE).

[41]  Niklas Sörensson,et al.  An Extensible SAT-solver , 2003, SAT.

[42]  Bernhard Rumpe,et al.  CDDiff: Semantic Differencing for Class Diagrams , 2011, ECOOP.

[43]  Martin Monperrus,et al.  Test case purification for improving fault localization , 2014, SIGSOFT FSE.

[44]  P. Jaccard,et al.  Etude comparative de la distribution florale dans une portion des Alpes et des Jura , 1901 .

[45]  Rupak Majumdar,et al.  Cause clue clauses: error localization using maximum satisfiability , 2010, PLDI '11.

[46]  AbreuRui,et al.  A practical evaluation of spectrum-based fault localization , 2009 .

[47]  William G. Griswold,et al.  Dynamically discovering likely program invariants to support program evolution , 1999, Proceedings of the 1999 International Conference on Software Engineering (IEEE Cat. No.99CB37002).

[48]  Sarfraz Khurshid,et al.  A Case for Efficient Solution Enumeration , 2003, SAT.

[49]  Yezid Donoso,et al.  Detecting Network Policy Conflicts Using Alloy , 2014, ABZ.

[50]  Marcelo F. Frias,et al.  TACO: Efficient SAT-Based Bounded Verification Using Symmetry Breaking and Tight Bounds , 2013, IEEE Transactions on Software Engineering.

[51]  Daniel J. Dougherty,et al.  The power of "why" and "why not": enriching scenario exploration with provenance , 2017, ESEC/SIGSOFT FSE.

[52]  Chao Liu,et al.  Statistical Debugging: A Hypothesis Testing-Based Approach , 2006, IEEE Transactions on Software Engineering.

[53]  Steven P. Reiss,et al.  Fault localization with nearest neighbor queries , 2003, 18th IEEE International Conference on Automated Software Engineering, 2003. Proceedings..

[54]  A.J.C. van Gemund,et al.  On the Accuracy of Spectrum-based Fault Localization , 2007, Testing: Academic and Industrial Conference Practice and Research Techniques - MUTATION (TAICPART-MUTATION 2007).

[55]  Shin Hong,et al.  Mutation-Based Fault Localization for Real-World Multilingual Programs (T) , 2015, 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE).

[56]  Sarfraz Khurshid,et al.  Learning to Optimize the Alloy Analyzer , 2019, 2019 12th IEEE Conference on Software Testing, Validation and Verification (ICST).

[57]  Yves Le Traon,et al.  Improving test suites for efficient fault localization , 2006, ICSE.

[58]  Mark Harman,et al.  Provably Optimal and Human-Competitive Results in SBSE for Spectrum Based Fault Localisation , 2013, SSBSE.

[59]  Andreas Zeller,et al.  Lightweight Defect Localization for Java , 2005, ECOOP.

[60]  Sarfraz Khurshid,et al.  TestEra: a novel framework for automated testing of Java programs , 2001, Proceedings 16th Annual International Conference on Automated Software Engineering (ASE 2001).

[61]  Kathi Fisler,et al.  Aluminum: Principled scenario exploration through minimality , 2013, 2013 35th International Conference on Software Engineering (ICSE).

[62]  Andreas Zeller,et al.  Yesterday, my program worked. Today, it does not. Why? , 1999, ESEC/FSE-7.

[63]  Xia Li,et al.  Boosting spectrum-based fault localization using PageRank , 2017, ISSTA.

[64]  Andreas Zeller,et al.  Locating causes of program failures , 2005, Proceedings. 27th International Conference on Software Engineering, 2005. ICSE 2005..

[65]  Westley Weimer,et al.  A human study of fault localization accuracy , 2010, 2010 IEEE International Conference on Software Maintenance.

[66]  Lee Naish,et al.  A model for spectra-based software diagnosis , 2011, TSEM.

[67]  Joseph Robert Horgan,et al.  Fault localization using execution slices and dataflow tests , 1995, Proceedings of Sixth International Symposium on Software Reliability Engineering. ISSRE'95.

[68]  Derek Rayside,et al.  Extending Alloy with Partial Instances , 2012, ABZ.

[69]  Sarfraz Khurshid,et al.  Solver-Based Sketching of Alloy Models Using Test Valuations , 2018, ABZ.

[70]  Michael D. Ernst,et al.  Evaluating and Improving Fault Localization , 2017, 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE).

[71]  Daniel Jackson,et al.  Alloy: a lightweight object modelling notation , 2002, TSEM.

[72]  Frank Tip,et al.  Directed test generation for effective fault localization , 2010, ISSTA '10.

[73]  Ilya Shlyakhter.  Generating effective symmetry-breaking predicates for search problems , 2007, Discret. Appl. Math..

[74]  Joseph Robert Horgan,et al.  Dynamic program slicing , 1990, PLDI '90.

[75]  Michael I. Jordan,et al.  Scalable statistical bug isolation , 2005, PLDI '05.

[76]  David Lo,et al.  Diversity maximization speedup for localizing faults in single-fault and multi-fault programs , 2016, Automated Software Engineering.

[77]  David Lo,et al.  Practitioners' expectations on automated fault localization , 2016, ISSTA.

[78]  Lars Grunske,et al.  A learning-to-rank based fault localization approach using likely invariants , 2016, ISSTA.

[79]  Sarfraz Khurshid,et al.  Incremental Analysis of Evolving Alloy Models , 2019, TACAS.