Testing bidirectional model transformation using metamorphic testing

Abstract Context In model-based software development, bidirectional model transformation (BX) provides a fundamental solution to model synchronization that can retain the consistency among models. Similar to conventional programs, a BX program may also contain bugs. Accordingly, a BX program must be tested prior to being used in practice. A challenging problem of testing BX programs is to construct test oracles (e.g., assertions and expected output models), which are usually difficult and/or expensive to manually specify. Objective In we paper, we investigate how to alleviate the oracle problem in BX testing via reducing the costs of developing test oracles. Method We propose a metamorphic testing approach for BX. First, we identify three generic metamorphic relations for BX. Afterwards, we define a metamodel MT4MT to establish metamorphic test groups and test scripts. We also propose a testing framework to support metamorphic testing based on MT4MT. Results We conducted an experimental study of mutation analysis and a case study on three ATL-based ad-hoc BXs. The results of the experimental study and the case study showed that our approach killed 79.38% mutants and enabled us to test real-world ATL-based ad-hoc BXs. We also demonstrated that MT4MT can be used to test the semantics properties of BXs. Conclusion Our approach is an effective and practical approach with lower costs of developing test oracles.

[1]  Benoit Baudry,et al.  Automatic Model Generation Strategies for Model Transformation Testing , 2009, ICMT@TOOLS.

[2]  Markus Scheidgen.  Generation of Large Random Models for Benchmarking , 2015, BigMDE@STAF.

[3]  Baowen Xu,et al.  Testing and validating machine learning classifiers by metamorphic testing , 2011, J. Syst. Softw..

[4]  Yves Le Traon,et al.  Qualifying input test data for model transformations , 2009, Software & Systems Modeling.

[5]  Dianfu Ma,et al.  From AADL to Timed Abstract State Machines: A verified model transformation , 2014, J. Syst. Softw..

[6]  Jon Whittle,et al.  Verifying semantic conformance of state machine-to-java code generators , 2010, MODELS'10.

[7]  Perdita Stevens,et al.  Bidirectional model transformations in QVT: semantic issues and open questions , 2007, MODELS'07.

[8]  Gabriele Taentzer,et al.  Proceedings of the 11th International Workshop on Graph Transformation and Visual Modeling Techniques (gt-vmt 2012) Instance Generation from Type Graphs with Arbitrary Multiplicities Instance Generation from Type Graphs with Arbitrary Multiplicities , 2022 .

[9]  Martin Gogolla,et al.  Testing models and model transformations using classifying terms , 2016, Software & Systems Modeling.

[10]  Zhiyi Ma,et al.  An MDE performance testing framework based on random model generation , 2016, J. Syst. Softw..

[11]  Jameleddine Hassine,et al.  Mutation Operators for the Atlas Transformation Language , 2013, 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation Workshops.

[12]  Huai Liu,et al.  A Metamorphic Relation-Based Approach to Testing Web Services Without Oracles , 2012, Int. J. Web Serv. Res..

[13]  Zhenjiang Hu,et al.  The essence of bidirectional programming , 2015, Science China Information Sciences.

[14]  Martin Gogolla,et al.  Tractable Model Transformation Testing , 2011, ECMFA.

[15]  Zhiyi Ma,et al.  A bidirectional-transformation-based framework for software visualization and visual editing , 2013, Science China Information Sciences.

[16]  Alexis Darrasse,et al.  Uniform Random Generation of Huge Metamodel Instances , 2009, ECMDA-FA.

[17]  Jean Bézivin,et al.  ATL: A model transformation tool , 2008, Sci. Comput. Program..

[18]  Jordi Cabot,et al.  Feature-based classification of bidirectional transformation approaches , 2014, Software & Systems Modeling.

[19]  Zhenjiang Hu,et al.  BiFluX: A Bidirectional Functional Update Language for XML , 2014, PPDP '14.

[20]  Gerti Kappel,et al.  Automated verification of model transformations based on visual contracts , 2013, Automated Software Engineering.

[21]  Yves Le Traon,et al.  Mutation Analysis Testing for Model Transformations , 2006, ECMDA-FA.

[22]  Yi Liu,et al.  Towards Parallel Model Generation for Random Performance Testing of Model-Oriented Operations , 2016, TASE.

[23]  Zhenjiang Hu,et al.  BiGUL: a formally verified core language for putback-based bidirectional programming , 2016, PEPM.

[24]  Horacio Hoyos,et al.  EMG: A Domain-Specific Transformation Language for Synthetic Model Generation , 2016, ICMT.

[25]  Hartmut Ehrig,et al.  From state- to delta-based bidirectional model transformations: the symmetric case , 2011, MODELS'11.

[26]  Jon Whittle,et al.  A survey of approaches for verifying model transformations , 2013, Software & Systems Modeling.

[27]  Richard F. Paige,et al.  EUnit: a unit testing framework for model management tasks , 2011, MODELS'11.

[28]  Benjamin C. Pierce,et al.  Combinators for bi-directional tree transformations: a linguistic approach to the view update problem , 2005, POPL '05.

[29]  Yi Liu,et al.  Towards Variability Management in Bidirectional Model Transformation , 2017, 2017 IEEE 41st Annual Computer Software and Applications Conference (COMPSAC).

[30]  Alcino Cunha,et al.  Implementing QVT-R Bidirectional Model Transformations Using Alloy , 2013, FASE.

[31]  Jürgen Dingel,et al.  Model transformation intents and their properties , 2016, Software & Systems Modeling.

[32]  Esther Guerra,et al.  Specification-driven model transformation testing , 2015, Software & Systems Modeling.

[33]  Gerson Sunyé,et al.  Partial Test Oracle in Model Transformation Testing , 2013, ICMT.

[34]  Huai Liu,et al.  An innovative approach for testing bioinformatics programs using metamorphic testing , 2009, BMC Bioinformatics.

[35]  Tsong Yueh Chen,et al.  Metamorphic Testing for Software Quality Assessment: A Study of Search Engines , 2016, IEEE Transactions on Software Engineering.

[36]  Kurt Stenzel,et al.  Formal verification of QVT transformations for code generation , 2011, MODELS'11.

[37]  Richard F. Paige,et al.  Model comparison: a foundation for model composition and model transformation testing , 2006, GaMMa '06.

[38]  Holger Giese,et al.  Towards Automatic Verification of Behavior Preservation for Model Transformation via Invariant Checking , 2012, ICGT.

[39]  Tsong Yueh Chen,et al.  Fault-based testing without the need of oracles , 2003, Inf. Softw. Technol..

[40]  Hartmut Ehrig,et al.  Model synchronization based on triple graph grammars: correctness, completeness and invertibility , 2012, Software & Systems Modeling.

[41]  Antonio Cicchetti,et al.  JTL: A Bidirectional and Change Propagating Transformation Language , 2010, SLE.

[42]  Tsong Yueh Chen,et al.  METRIC: METamorphic Relation Identification based on the Category-choice framework , 2016, J. Syst. Softw..

[43]  Manuel Wimmer,et al.  Fully verifying transformation contracts for declarative ATL , 2015, 2015 ACM/IEEE 18th International Conference on Model Driven Engineering Languages and Systems (MODELS).

[44]  Thomas Vogel,et al.  Incremental model synchronization for efficient run-time monitoring , 2009, MODELS'09.

[45]  Jeffrey M. Voas,et al.  Metamorphic Testing for Cybersecurity , 2016, Computer.

[46]  Stephen S. Yau,et al.  Integration Testing of Context-sensitive Middleware-based Applications: a Metamorphic Approach , 2006, Int. J. Softw. Eng. Knowl. Eng..

[47]  Marjan Mernik,et al.  Converting metamodels to graph grammars: doing without advanced graph grammar features , 2013, Software & Systems Modeling.

[48]  Lu Zhang,et al.  Search-based inference of polynomial metamorphic relations , 2014, ASE.

[49]  Jordi Cabot,et al.  ATLTest: a white-box test generation approach for ATL transformations , 2012, MODELS'12.

[50]  Jan Oliver Ringert,et al.  A framework for relating syntactic and semantic model differences , 2015, MoDELS.

[51]  Shinya Kawanaka,et al.  biXid: a bidirectional transformation language for XML , 2006, ICFP '06.

[52]  Tsong Yueh Chen,et al.  Metamorphic testing of programs on partial differential equations: a case study , 2002, Proceedings 26th Annual International Computer Software and Applications.

[53]  Holger Giese,et al.  From model transformation to incremental bidirectional model synchronization , 2009, Software & Systems Modeling.

[54]  Sergio Segura,et al.  Automated inference of likely metamorphic relations for model transformations , 2018, J. Syst. Softw..

[55]  Martin Gogolla,et al.  Validating UML and OCL models in USE by automatic snapshot generation , 2005, Software & Systems Modeling.

[56]  Andy Schürr,et al.  On the Usage of TGGs for Automated Model Transformation Testing , 2014, ICMT.

[57]  Manuel Wimmer,et al.  Towards systematic mutations for and with ATL model transformations , 2015, 2015 IEEE Eighth International Conference on Software Testing, Verification and Validation Workshops (ICSTW).

[58]  Hui Song,et al.  Supporting runtime software architecture: A bidirectional-transformation-based approach , 2011, J. Syst. Softw..

[59]  Andy Schürr,et al.  Bidirectional Model Transformation with Precedence Triple Graph Grammars , 2012, ECMFA.

[60]  Sergio Segura,et al.  A Survey on Metamorphic Testing , 2016, IEEE Transactions on Software Engineering.

[61]  Hartmut Ehrig,et al.  Semantical Correctness and Completeness of Model Transformations Using Graph and Rule Transformation , 2008, ICGT.

[62]  Krzysztof Czarnecki,et al.  A three-dimensional taxonomy for bidirectional model synchronization , 2016, J. Syst. Softw..

[63]  Yves Le Traon,et al.  Model transformation testing: oracle issue , 2008, 2008 IEEE International Conference on Software Testing Verification and Validation Workshop.

[64]  Thomas Baar,et al.  A Graphical Approach to Prove the Semantic Preservation of UML/OCL Refactoring Rules , 2006, Ershov Memorial Conference.

[65]  Gabriele Taentzer,et al.  Generating instance models from meta models , 2006, Software & Systems Modeling.

[66]  Jan Ladleif,et al.  Model-driven test case design for model-to-model semantics preservation , 2015, A-TEST@SIGSOFT FSE.

[67]  Linpeng Huang,et al.  A Bidirectional Framework for Model Synchronization in Component-Based Software Development , 2013, 2013 13th International Conference on Quality Software.

[68]  Zuohua Ding,et al.  Testing Model Transformation Programs using Metamorphic Testing , 2014, SEKE.

[69]  Jim Steel,et al.  Metamodel-based Test Generation for Model Transformations: an Algorithm and a Tool , 2006, 2006 17th International Symposium on Software Reliability Engineering.

[70]  Huai Liu,et al.  How Effectively Does Metamorphic Testing Alleviate the Oracle Problem? , 2014, IEEE Transactions on Software Engineering.

[71]  Hui Song,et al.  Synchronizing concurrent model updates based on bidirectional transformation , 2013, Software & Systems Modeling.

[72]  Zinovy Diskin,et al.  Algebraic Models for Bidirectional Model Synchronization , 2008, MoDELS.

[73]  Jürgen Dingel,et al.  Model transformation testing: the state of the art , 2012, AMT '12.

[74]  He Xiao,et al.  Randomized Model Generation for Performance Testing of Model Transformations , 2014, 2014 IEEE 38th Annual Computer Software and Applications Conference.

[75]  Felix T.S. Chan,et al.  Application of metamorphic testing in numerical analysis , 1998, ICSE 1998.