Spectrum-Based Fault Localization in Model Transformations

Model transformations play a cornerstone role in Model-Driven Engineering (MDE), as they provide the essential mechanisms for manipulating and transforming models. The correctness of software built using MDE techniques greatly relies on the correctness of model transformations. However, it is challenging and error prone to debug them, and the situation gets more critical as the size and complexity of model transformations grow, where manual debugging is no longer possible. Spectrum-Based Fault Localization (SBFL) uses the results of test cases and their corresponding code coverage information to estimate the likelihood of each program component (e.g., statements) of being faulty. In this article we present an approach to apply SBFL for locating the faulty rules in model transformations. We evaluate the feasibility and accuracy of the approach by comparing the effectiveness of 18 different state-of-the-art SBFL techniques at locating faults in model transformations. Evaluation results revealed that the best techniques, namely Kulcynski2, Mountford, Ochiai, and Zoltar, lead the debugger to inspect a maximum of three rules to locate the bug in around 74% of the cases. Furthermore, we compare our approach with a static approach for fault localization in model transformations, observing a clear superiority of the proposed SBFL-based method.

[1]  Mary Jean Harrold,et al.  An empirical study of the effects of test-suite reduction on fault localization , 2008, 2008 ACM/IEEE 30th International Conference on Software Engineering.

[2]  David Lo,et al.  Are faults localizable? , 2012, 2012 9th IEEE Working Conference on Mining Software Repositories (MSR).

[3]  L. Darrell Whitley,et al.  Subtle higher order mutants , 2017, Inf. Softw. Technol..

[4]  Gerti Kappel,et al.  A Petri Net Based Debugging Environment for QVT Relations , 2009, 2009 IEEE/ACM International Conference on Automated Software Engineering.

[5]  Manuel Wimmer,et al.  Full contract verification for ATL using symbolic execution , 2016, Software & Systems Modeling.

[6]  J. Troya,et al.  Marrying Search-based Optimization and Model Transformation Technology , 2015 .

[7]  Jochen Ludewig,et al.  Models in software engineering - an introduction , 2004, Informatik Forschung und Entwicklung.

[8]  Lionel C. Briand,et al.  Generating Test Data from OCL Constraints with Search Techniques , 2013, IEEE Transactions on Software Engineering.

[9]  Laurence Duchien,et al.  OCL for the Specification of Model Transformation Contracts , 2004 .

[10]  Juan de Lara,et al.  Static Analysis of Model Transformations , 2017, IEEE Transactions on Software Engineering.

[11]  K. Rustan M. Leino,et al.  This is Boogie 2 , 2016 .

[12]  Bernhard Schätz,et al.  Graph and model transformation tools for model migration , 2014, Software & Systems Modeling.

[13]  Zheng Cheng,et al.  A Deductive Approach for Fault Localization in ATL Model Transformations , 2017, FASE.

[14]  Antonio Ruiz Cortés,et al.  An Approach for Debugging Model Transformations Applying Spectrum-Based Fault Localization , 2017 .

[15]  W. Eric Wong,et al.  Software Fault Localization Using DStar (D*) , 2012, 2012 IEEE Sixth International Conference on Software Security and Reliability.

[16]  Rui Abreu,et al.  Exploiting count spectra for Bayesian fault localization , 2010, PROMISE '10.

[17]  Juan de Lara,et al.  Translating Target to Source Constraints in Model-to-Model Transformations , 2017, 2017 ACM/IEEE 20th International Conference on Model Driven Engineering Languages and Systems (MODELS).

[18]  Manuel Wimmer,et al.  Towards systematic mutations for and with ATL model transformations , 2015, 2015 IEEE Eighth International Conference on Software Testing, Verification and Validation Workshops (ICSTW).

[19]  Esperanza Marcos,et al.  MeTAGeM-Trace: Improving trace generation in model transformation by leveraging the role of transformation models , 2015, Sci. Comput. Program..

[20]  Rui Abreu,et al.  A Survey on Software Fault Localization , 2016, IEEE Transactions on Software Engineering.

[21]  T. H. Tse,et al.  Is non-parametric hypothesis testing model robust for statistical fault localization? , 2009, Inf. Softw. Technol..

[22]  Antonio Vallecillo,et al.  A Rewriting Logic Semantics for ATL , 2011, J. Object Technol..

[23]  Shin Yoo,et al.  Evolving Human Competitive Spectra-Based Fault Localisation Techniques , 2012, SSBSE.

[24]  James F. Power,et al.  A Sound Execution Semantics for ATL via Translation Validation - Research Paper , 2015, ICMT.

[25]  Jürgen Dingel,et al.  Model transformation intents and their properties , 2016, Software & Systems Modeling.

[26]  Ekkart Kindler,et al.  Comparing relational model transformation technologies: implementing Query/View/Transformation with Triple Graph Grammars , 2009, Software & Systems Modeling.

[27]  Rui Abreu,et al.  Zoltar: a spectrum-based fault localization tool , 2009, SINTER '09.

[28]  Lee Naish,et al.  A model for spectra-based software diagnosis , 2011, TSEM.

[29]  M. Friedman.  A Comparison of Alternative Tests of Significance for the Problem of $m$ Rankings , 1940 .

[30]  Jim Steel,et al.  Metamodel-based Test Generation for Model Transformations: an Algorithm and a Tool , 2006, 2006 17th International Symposium on Software Reliability Engineering.

[31]  Anneke Kleppe,et al.  The Object Constraint Language: Getting Your Models Ready for MDA , 2003 .

[32]  W. Eric Wong,et al.  The DStar Method for Effective Software Fault Localization , 2014, IEEE Transactions on Reliability.

[33]  Mark Harman,et al.  The Oracle Problem in Software Testing: A Survey , 2015, IEEE Transactions on Software Engineering.

[34]  Claes Wohlin,et al.  Experimentation in Software Engineering , 2000, The Kluwer International Series in Software Engineering.

[35]  Gabriele Taentzer,et al.  AGG: A Graph Transformation Environment for Modeling and Validation of Software , 2003, AGTIVE.

[36]  Baowen Xu,et al.  On the analysis of spectrum based fault localization using hitting sets , 2019, J. Syst. Softw..

[37]  Dániel Varró,et al.  A research roadmap towards achieving scalability in model driven engineering , 2013, BigMDE '13.

[38]  Nora Szasz,et al.  A Type-Theoretic Framework for Certified Model Transformations , 2010, SBMF.

[39]  Norman J. Walsh,et al.  DocBook: The Definitive Guide , 1999 .

[40]  Mark Harman,et al.  Higher Order Mutation Testing , 2009, Inf. Softw. Technol..

[41]  Clémentine Nebut,et al.  Towards a Traceability Framework for Model Transformations in Kermeta , 2006 .

[42]  Esther Guerra,et al.  Specification-driven model transformation testing , 2015, Software & Systems Modeling.

[43]  Gregg Rothermel,et al.  An empirical investigation of the relationship between spectra differences and regression faults , 2000 .

[44]  S. Holm.  A Simple Sequentially Rejective Multiple Test Procedure , 1979 .

[45]  Antonio Vallecillo,et al.  A graphical approach for modeling time-dependent behavior of DSLs , 2009, 2009 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC).

[46]  Andrea Arcuri,et al.  Improved heuristics for solving OCL constraints using search algorithms , 2014, GECCO.

[47]  Indrakshi Ray,et al.  On challenges of model transformation from UML to Alloy , 2008, Software & Systems Modeling.

[48]  Juan de Lara,et al.  Quick fixing ATL transformations with speculative analysis , 2016, Software & Systems Modeling.

[49]  Esther Guerra,et al.  Specification-Driven Test Generation for Model Transformations , 2012, ICMT@TOOLS.

[50]  Mary Jean Harrold,et al.  Empirical evaluation of the tarantula automatic fault-localization technique , 2005, ASE.

[51]  James M. Bieman,et al.  Fault localization for automated program repair: effectiveness, performance, repair correctness , 2017, Software Quality Journal.

[52]  Lee Naish,et al.  Effective Software Bug Localization Using Spectral Frequency Weighting Function , 2010, 2010 IEEE 34th Annual Computer Software and Applications Conference.

[53]  David Lo,et al.  Extended comprehensive study of association measures for fault localization , 2014, J. Softw. Evol. Process..

[54]  Gabriele Taentzer,et al.  Henshin: advanced concepts and tools for in-place EMF model transformations , 2010, MODELS'10.

[55]  Jesús Sánchez Cuadrado,et al.  Modularization of model transformations through a phasing mechanism , 2009, Software & Systems Modeling.

[56]  Yves Le Traon,et al.  Model Transformation Testing Challenges , 2006 .

[57]  A. E. Maxwell,et al.  Deriving coefficients of reliability and agreement for ratings. , 1968, The British journal of mathematical and statistical psychology.

[58]  Chao Liu,et al.  Statistical Debugging: A Hypothesis Testing-Based Approach , 2006, IEEE Transactions on Software Engineering.

[59]  Manuel Wimmer,et al.  Search-Based Model Transformations with MOMoT , 2016, ICMT.

[60]  Juan de Lara,et al.  AToM3: A Tool for Multi-formalism and Meta-modelling , 2002, FASE.

[61]  Jordi Cabot,et al.  Model-Driven Software Engineering in Practice , 2017, Synthesis Lectures on Software Engineering.

[62]  Hans Vangheluwe,et al.  A characterization of integrated multi-view modeling in the context of embedded and cyber-physical systems , 2013, 2013 Proceedings of the International Conference on Embedded Software (EMSOFT).

[63]  Lee Naish,et al.  Multiple Bug Spectral Fault Localization Using Genetic Programming , 2015, 2015 24th Australasian Software Engineering Conference.

[64]  Yves Le Traon,et al.  Mutation Analysis Testing for Model Transformations , 2006, ECMDA-FA.

[65]  Manuel Wimmer,et al.  Testing M2T/T2M Transformations , 2013, MoDELS.

[66]  Steffen Zschaler,et al.  Modular DSLs for Flexible Analysis: An e-Motions Reimplementation of Palladio , 2014, ECMFA.

[67]  Vicente Pelechano,et al.  Test-Driven Development of Model Transformations , 2009, MoDELS.

[68]  Shane Sendall,et al.  Model Transformation: The Heart and Soul of Model-Driven Software Development , 2003, IEEE Softw..

[69]  Olivier Barais,et al.  Model Driven Language Engineering with Kermeta , 2009, GTTSE.

[70]  Francisco Herrera,et al.  A practical tutorial on the use of nonparametric statistical tests as a methodology for comparing evolutionary and swarm intelligence algorithms , 2011, Swarm Evol. Comput..

[71]  Iman Poernomo,et al.  Correct-by-Construction Model Transformations from Partially Ordered Specifications in Coq , 2010, ICFEM.

[72]  Dániel Varró,et al.  VIATRA - visual automated transformations for formal verification and validation of UML models , 2002, Proceedings 17th IEEE International Conference on Automated Software Engineering,.

[73]  Jordi Cabot,et al.  ATLTest: a white-box test generation approach for ATL transformations , 2012, MODELS'12.

[74]  Narciso Martí-Oliet,et al.  All About Maude - A High-Performance Logical Framework, How to Specify, Program and Verify Systems in Rewriting Logic , 2007, All About Maude.

[75]  Akbar Siami Namin,et al.  How Significant is the Effect of Fault Interactions on Coverage-Based Fault Localizations? , 2013, 2013 ACM / IEEE International Symposium on Empirical Software Engineering and Measurement.

[76]  Benoit Baudry,et al.  Automatic Model Generation Strategies for Model Transformation Testing , 2009, ICMT@TOOLS.

[77]  Thomas Kühne,et al.  Matters of (Meta-) Modeling , 2006, Software & Systems Modeling.

[78]  Manuel Wimmer,et al.  Search‐based model transformations , 2016, J. Softw. Evol. Process..

[79]  Yves Le Traon,et al.  Qualifying input test data for model transformations , 2009, Software & Systems Modeling.

[80]  Xiangyu Zhang,et al.  Towards locating execution omission errors , 2007, PLDI '07.

[81]  Antonio Vallecillo,et al.  Static Fault Localization in Model Transformations , 2015, IEEE Transactions on Software Engineering.

[82]  Gerti Kappel,et al.  Automated verification of model transformations based on visual contracts , 2013, Automated Software Engineering.

[83]  Peter Zoeteweij,et al.  Spectrum-Based Multiple Fault Localization , 2009, 2009 IEEE/ACM International Conference on Automated Software Engineering.

[84]  Yuhua Qi,et al.  Using automated program repair for evaluating the effectiveness of fault localization techniques , 2013, ISSTA.

[85]  Jesús Manuel Almendros-Jiménez,et al.  Automatic Generation of Ecore Models for Testing ATL Transformations , 2016, MEDI.

[86]  Krzysztof Czarnecki,et al.  Feature-based survey of model transformation approaches , 2006, IBM Syst. J..

[87]  Peter Zoeteweij,et al.  A practical evaluation of spectrum-based fault localization , 2009, J. Syst. Softw..

[88]  Jürgen Dingel,et al.  Model Transformations for Migrating Legacy Models: An Industrial Case Study , 2012, ECMFA.

[89]  Jordi Cabot,et al.  A Catalogue of Refactorings for Model-to-Model Transformations , 2012, J. Object Technol..

[90]  Jean-Luc Dekeyser,et al.  Towards an automation of the mutation analysis dedicated to model transformation , 2015, Softw. Test. Verification Reliab..

[91]  Baowen Xu,et al.  A theoretical analysis of the risk evaluation formulas for spectrum-based fault localization , 2013, TSEM.

[92]  Alberto Rodrigues da Silva,et al.  Model-driven engineering: A survey supported by the unified conceptual model , 2015, Comput. Lang. Syst. Struct..

[93]  Antonio Vallecillo,et al.  Specification and simulation of queuing network models using Domain-Specific Languages , 2014, Comput. Stand. Interfaces.

[94]  Stephen J. Mellor,et al.  MDA Distilled Principles Of Model-Driven Architecture , 2004 .

[95]  Ying Zhang,et al.  Testing bidirectional model transformation using metamorphic testing , 2018, Inf. Softw. Technol..

[96]  Martin Gogolla,et al.  Tractable Model Transformation Testing , 2011, ECMFA.

[97]  Jean Bézivin,et al.  ATL: A model transformation tool , 2008, Sci. Comput. Program..

[98]  Marcos Didonet Del Fabro,et al.  Transforming Very Large Models in the Cloud: a Research Roadmap , 2012 .

[99]  Antonio Cicchetti,et al.  JTL: A Bidirectional and Change Propagating Transformation Language , 2010, SLE.

[100]  Mark Harman,et al.  An Analysis and Survey of the Development of Mutation Testing , 2011, IEEE Transactions on Software Engineering.

[101]  Martin Gogolla,et al.  Typing Model Transformations Using Tracts , 2012, ICMT@TOOLS.

[102]  Yuhua Qi,et al.  Slice-based statistical fault localization , 2014, J. Syst. Softw..

[103]  K. Goseva-Popstojanova,et al.  Common Trends in Software Fault and Failure Data , 2009, IEEE Transactions on Software Engineering.

[104]  Jean Bézivin,et al.  Building DSLs with AMMA/ATL, a Case Study on SPL and CPL Telephony Languages , 2006 .

[105]  A. Vargha,et al.  A Critique and Improvement of the CL Common Language Effect Size Statistics of McGraw and Wong , 2000 .

[106]  Marouane Kessentini,et al.  Model Transformation Modularization as a Many-Objective Optimization Problem , 2017, IEEE Transactions on Software Engineering.

[107]  Sergio Segura,et al.  Automated inference of likely metamorphic relations for model transformations , 2018, J. Syst. Softw..

[108]  Jordi Cabot,et al.  Distributed model-to-model transformation with ATL on MapReduce , 2015, SLE.

[109]  Jean Bézivin,et al.  ATL: a QVT-like transformation language , 2006, OOPSLA '06.

[110]  A.J.C. van Gemund,et al.  On the Accuracy of Spectrum-based Fault Localization , 2007, Testing: Academic and Industrial Conference Practice and Research Techniques - MUTATION (TAICPART-MUTATION 2007).

[111]  Francisco Durán,et al.  On the Reusable Specification of Non-functional Properties in DSLs , 2012, SLE.

[112]  Wei Li,et al.  Effects of Class Imbalance in Test Suites: An Empirical Study of Spectrum-Based Fault Localization , 2012, 2012 IEEE 36th Annual Computer Software and Applications Conference Workshops.

[113]  Frédéric Jouault.  Loosely Coupled Traceability for ATL , 2005 .

[114]  Philip H. Ramsey.  Nonparametric Statistical Methods , 1974, Technometrics.

[115]  Sergio Segura,et al.  A Survey on Metamorphic Testing , 2016, IEEE Transactions on Software Engineering.