Metamorphic Relations for Enhancing System Understanding and Use

Modern information technology paradigms, such as online services and off-the-shelf products, often involve a wide variety of users with different or even conflicting objectives. Every software output may satisfy some users, but may also fail to satisfy others. Furthermore, users often do not know the internal working mechanisms of the systems. This situation is quite different from bespoke software, where developers and users typically know each other. This paper proposes an approach to help users to better understand the software that they use, and thereby more easily achieve their objectives—even when they do not fully understand how the system is implemented. Our approach borrows the concept of metamorphic relations from the field of metamorphic testing (MT), using it in an innovative way that extends beyond MT. We also propose a “symmetry” metamorphic relation pattern and a “change direction” metamorphic relation input pattern that can be used to derive multiple concrete metamorphic relations. Empirical studies reveal previously unknown failures in some of the most popular applications in the world, and show how our approach can help users to better understand and better use the systems. The empirical results provide strong evidence of the simplicity, applicability, and effectiveness of our methodology.

[1]  Upulee Kanewala,et al.  Automated Test Oracles: State of the Art, Taxonomies, and Trends , 2015, Adv. Comput..

[2]  Tsong Yueh Chen,et al.  Automated functional testing of web search engines in the absence of an oracle , 2007 .

[3]  Tsong Yueh Chen,et al.  Metamorphic Testing for Adobe Data Analytics Software , 2017, 2017 IEEE/ACM 2nd International Workshop on Metamorphic Testing (MET).

[4]  Hasan Sözer,et al.  VISOR: A fast image processing pipeline with scaling and translation invariance for test oracle automation of visual output systems , 2018, J. Syst. Softw..

[5]  Baowen Xu,et al.  Testing and validating machine learning classifiers by metamorphic testing , 2011, J. Syst. Softw..

[6]  Mike Chiasson,et al.  Executable acceptance tests for communicating business requirements: customer perspective , 2006, AGILE 2006 (AGILE'06).

[7]  Tsong Yueh Chen,et al.  Fault-based testing without the need of oracles , 2003, Inf. Softw. Technol..

[8]  Charles Alan Wright,et al.  A Short History of the Shadow , 1997 .

[9]  Bogdan Korel Black-box understanding of COTS components , 1999, Proceedings Seventh International Workshop on Program Comprehension.

[10]  D. Perrett,et al.  Symmetry and human facial attractiveness. , 1999 .

[11]  P. Shekelle,et al.  Web appendix A , 2013 .

[12]  Tsong Yueh Chen,et al.  On the Correlation between the Effectiveness of Metamorphic Relations and Dissimilarities of Test Case Executions , 2013, 2013 13th International Conference on Quality Software.

[13]  Mikael Lindvall,et al.  Metamorphic Model-Based Testing Applied on NASA DAT -- An Experience Report , 2015, 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering.

[14]  Tsong Yueh Chen,et al.  Automated functional testing of online search services , 2012, Softw. Test. Verification Reliab..

[15]  Arnaud Gotlieb Exploiting symmetries to test programs , 2003, 14th International Symposium on Software Reliability Engineering, 2003. ISSRE 2003..

[16]  Ian Stewart Symmetry: A Very Short Introduction , 2013 .

[17]  Paul Ammann,et al.  Data Diversity: An Approach to Software Fault Tolerance , 1988, IEEE Trans. Computers.

[18]  Tsong Yueh Chen,et al.  Metamorphic Testing for Software Quality Assessment: A Study of Search Engines , 2016, IEEE Transactions on Software Engineering.

[19]  Tsong Yueh Chen,et al.  Metamorphic Testing: A New Approach for Generating Next Test Cases , 2020, ArXiv.

[20]  M'aximo Banados,et al.  A short review on Noether's theorems, gauge symmetries and boundary terms, for students , 2016, 1601.03616.

[21]  ISO / IEC 25010 : 2011 Systems and software engineering — Systems and software Quality Requirements and Evaluation ( SQuaRE ) — System and software quality models , 2013 .

[22]  D. Gross,et al.  The role of symmetry in fundamental physics. , 1996, Proceedings of the National Academy of Sciences of the United States of America.

[23]  Kent L. Beck,et al.  Extreme programming explained - embrace change , 1990 .

[24]  Sergio Segura,et al.  Automated inference of likely metamorphic relations for model transformations , 2018, J. Syst. Softw..

[25]  Yongsheng Gao,et al.  Recognizing Rotated Faces From Frontal and Side Views: An Approach Toward Effective Use of Mugshot Databases , 2008, IEEE Transactions on Information Forensics and Security.

[26]  A. Rangaswamy,et al.  Consumer Choice Behavior in Online and Traditional Supermarkets: The Effects of Brand Name, Price, , 2000 .

[27]  P. Anderson More is different. , 1972, Science.

[28]  István Hargittai,et al.  Symmetry through the eyes of a chemist , 1986 .

[29]  James M. Bieman,et al.  Predicting metamorphic relations for testing scientific software: a machine learning approach using graph kernels , 2016, Softw. Test. Verification Reliab..

[30]  M.M. Lehman,et al.  Programs, life cycles, and laws of software evolution , 1980, Proceedings of the IEEE.

[31]  Gordon Fraser,et al.  CrowdOracles: Can the Crowd Solve the Oracle Problem? , 2013, 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation.

[32]  Sergio Segura,et al.  A Survey on Metamorphic Testing , 2016, IEEE Transactions on Software Engineering.

[33]  Dave Towey,et al.  A revisit of three studies related to random testing , 2015, Science China Information Sciences.

[34]  Zhenchang Xing,et al.  Measuring Program Comprehension: A Large-Scale Field Study with Professionals , 2018, IEEE Transactions on Software Engineering.

[35]  Srinivas Aluru,et al.  Message from the workshop chairs , 2009 .

[36]  Bernhard Koerber,et al.  IT works , 2006, LOG IN.

[37]  宮下 誠 書評 Victor I.Stoichita, A Short History of the Shadow (特集 パレルゴン:美術における付随的なもの) , 2003 .

[38]  Huai Liu,et al.  How Effectively Does Metamorphic Testing Alleviate the Oracle Problem? , 2014, IEEE Transactions on Software Engineering.

[39]  D. Richard Kuhn,et al.  Finding Bugs in Cryptographic Hash Function Implementations , 2018, IEEE Transactions on Reliability.

[40]  Sudipto Ghosh,et al.  A model for understanding software components , 2002, International Conference on Software Maintenance, 2002. Proceedings..

[41]  THE NEW A vision and R & D update from Accenture Labs and Accenture Testing Services , .

[42]  Alastair F. Donaldson,et al.  Automated testing of graphics shader compilers , 2017, Proc. ACM Program. Lang..

[43]  Dave Towey,et al.  Metamorphic testing: A new student engagement approach for a new software testing paradigm , 2016, 2016 IEEE International Conference on Teaching, Assessment, and Learning for Engineering (TALE).

[44]  Frank E. Harris,et al.  Mathematical Methods for Physicists: A Comprehensive Guide , 2012 .

[45]  David A. Bader,et al.  Message from the workshop chairs , 2010, 2010 IEEE International Symposium on Parallel & Distributed Processing, Workshops and Phd Forum (IPDPSW).

[46]  Hong Zhu,et al.  JFuzz: A Tool for Automated Java Unit Testing Based on Data Mutation and Metamorphic Testing Methods , 2015, 2015 Second International Conference on Trustworthy Systems and Their Applications.

[47]  H. Kragh Contemporary History of Cosmology and the Controversy over the Multiverse , 2009 .

[48]  David L. Dill,et al.  Better verification through symmetry , 1996, Formal Methods Syst. Des..

[49]  Hong Zhu,et al.  Generating Structurally Complex Test Cases By Data Mutation: A Case Study Of Testing An Automated Modelling Tool , 2009, Comput. J..

[50]  Alastair F. Donaldson,et al.  Metamorphic Testing for (Graphics) Compilers , 2016, 2016 IEEE/ACM 1st International Workshop on Metamorphic Testing (MET).

[51]  John M. Rose,et al.  Interactive stated choice surveys: a study of air travel behaviour , 2012 .

[52]  Alessandra Gorla,et al.  Automatic Workarounds: Exploiting the Intrinsic Redundancy of Web Applications , 2015, TSEM.

[53]  Arie van Deursen,et al.  A Systematic Survey of Program Comprehension through Dynamic Analysis , 2008, IEEE Transactions on Software Engineering.

[54]  Collin McMillan,et al.  Detecting User Story Information in Developer-Client Conversations to Generate Extractive Summaries , 2017, 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE).

[55]  Yang-Wai Chow,et al.  Metamorphic Testing of Navigation Software: A Pilot Study with Google Maps , 2018, HICSS.

[56]  Suman Jana,et al.  DeepTest: Automated Testing of Deep-Neural-Network-Driven Autonomous Cars , 2017, 2018 IEEE/ACM 40th International Conference on Software Engineering (ICSE).

[57]  Dave Towey,et al.  Teaching software testing skills: Metamorphic testing as vehicle for creativity and effectiveness in software testing , 2015, 2015 IEEE International Conference on Teaching, Assessment, and Learning for Engineering (TALE).

[58]  Tsong Yueh Chen,et al.  Case studies on the selection of useful relations in metamorphic testing , 2004 .

[59]  Howard Barnum,et al.  The Beginning of Infinity: Explanations That Transform the World , 2012 .

[60]  Mark Harman,et al.  A survey of the use of crowdsourcing in software engineering , 2017, J. Syst. Softw..

[61]  Dietmar P. F. Möller,et al.  Systems and Software Engineering , 2016 .

[62]  P. Kaboš,et al.  Magnetostatic waves and their application , 1993 .

[63]  Cheng Zhang,et al.  Automated Test Oracles: A Survey , 2015, Adv. Comput..

[64]  A. Zee,et al.  Fearful Symmetry: The Search for Beauty in Modern Physics , 1986 .

[65]  Zhendong Su,et al.  Compiler validation via equivalence modulo inputs , 2014, PLDI.

[66]  Jeffrey M. Voas,et al.  Metamorphic Testing for Cybersecurity , 2016, Computer.

[67]  Christoph Schulze,et al.  Metamorphic Model-Based Testing of Autonomous Systems , 2017, 2017 IEEE/ACM 2nd International Workshop on Metamorphic Testing (MET).

[68]  Mark Harman,et al.  The Oracle Problem in Software Testing: A Survey , 2015, IEEE Transactions on Software Engineering.

[69]  Ė. B. Vinberg,et al.  A course in algebra , 2003 .

[70]  Gabor Kunstatter,et al.  The central role of symmetry in physics , 2016 .

[71]  R. P. Jagadeesh Chandra Bose,et al.  Identifying implementation bugs in machine learning based image classifiers using metamorphic testing , 2018, ISSTA.

[72]  Gail E. Kaiser,et al.  Dynamic Inference of Likely Metamorphic Properties to Support Differential Testing , 2015, 2015 IEEE/ACM 10th International Workshop on Automation of Software Test.

[73]  Kent Beck,et al.  Extreme Programming Explained: Embrace Change (2nd Edition) , 2004 .

[74]  Sergio Segura,et al.  Metamorphic Testing of RESTful Web APIs , 2018, IEEE Transactions on Software Engineering.