The Impact of a Dot: Case Studies of a Noise Metamorphic Relation Pattern

We propose a "noise" metamorphic relation pattern (MRP), which is a sub-pattern under the more general MRP "symmetry." We conduct case studies with real-life systems in three different application domains (obstacle perception in autonomous systems, machine translation, and named entity recognition) to show the usefulness of the "noise" MRP for software verification and validation.

[1]  Madhusudan Srinivasan.  Prioritization of Metamorphic Relations Based on Test Case Execution Properties , 2018, 2018 IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW).

[2]  Sergio Segura,et al.  A Survey on Metamorphic Testing , 2016, IEEE Transactions on Software Engineering.

[3]  Huai Liu,et al.  Metamorphic Testing , 2018, ACM Comput. Surv..

[4]  P. Anderson.  More is different. , 1972, Science.

[5]  Tsong Yueh Chen,et al.  Automated functional testing of web search engines in the absence of an oracle , 2007 .

[6]  Carlos E. Otero,et al.  Research Directions for Engineering Big Data Analytics Software , 2015, IEEE Intelligent Systems.

[7]  Zhenyu Wang,et al.  Metamorphic Testing for Adobe Analytics Data Collection JavaScript Library , 2018, 2018 IEEE/ACM 3rd International Workshop on Metamorphic Testing (MET).

[8]  Tsong Yueh Chen,et al.  An effective testing method for end-user programmers , 2005, WEUSE@ICSE.

[9]  Dave Towey,et al.  Introduction to the special issue on test oracles , 2018, J. Syst. Softw..

[10]  Tsong Yueh Chen,et al.  Fault-based testing without the need of oracles , 2003, Inf. Softw. Technol..

[11]  Baowen Xu,et al.  Testing and validating machine learning classifiers by metamorphic testing , 2011, J. Syst. Softw..

[12]  Tsong Yueh Chen,et al.  Automated functional testing of online search services , 2012, Softw. Test. Verification Reliab..

[13]  Christoph Schulze,et al.  Metamorphic Model-Based Testing of Autonomous Systems , 2017, 2017 IEEE/ACM 2nd International Workshop on Metamorphic Testing (MET).

[14]  Mark Harman,et al.  The Oracle Problem in Software Testing: A Survey , 2015, IEEE Transactions on Software Engineering.

[15]  Suman Jana,et al.  DeepTest: Automated Testing of Deep-Neural-Network-Driven Autonomous Cars , 2017, 2018 IEEE/ACM 40th International Conference on Software Engineering (ICSE).

[16]  R. P. Jagadeesh Chandra Bose,et al.  Identifying implementation bugs in machine learning based image classifiers using metamorphic testing , 2018, ISSTA.

[17]  Sergio Segura,et al.  Metamorphic Testing of RESTful Web APIs , 2018, IEEE Transactions on Software Engineering.

[18]  Hironori Washizaki,et al.  Abstract security patterns , 2008 .

[19]  Zhi Quan Zhou,et al.  Metamorphic Testing for Machine Translations: MT4MT , 2018, 2018 25th Australasian Software Engineering Conference (ASWEC).

[20]  Sarfraz Khurshid,et al.  DeepRoad: GAN-Based Metamorphic Testing and Input Validation Framework for Autonomous Driving Systems , 2018, 2018 33rd IEEE/ACM International Conference on Automated Software Engineering (ASE).

[21]  Liqun Sun,et al.  Metamorphic testing of driverless cars , 2019, Commun. ACM.

[22]  Dave Towey,et al.  Metamorphic Relations for Enhancing System Understanding and Use , 2020, IEEE Transactions on Software Engineering.

[23]  Tsong Yueh Chen,et al.  Metamorphic Testing: A New Approach for Generating Next Test Cases , 2020, ArXiv.

[24]  Dave Towey,et al.  A Monte Carlo Method for Metamorphic Testing of Machine Translation Services , 2018, 2018 IEEE/ACM 3rd International Workshop on Metamorphic Testing (MET).

[25]  Tsong Yueh Chen,et al.  Metamorphic Testing for Software Quality Assessment: A Study of Search Engines , 2016, IEEE Transactions on Software Engineering.

[26]  M. Govindan,et al.  Purification of two dexamethasone-binding proteins from rat-liver cytosol. , 1978, European journal of biochemistry.

[27]  The Efficacy of Round-trip Translation for MT Evaluation , 2010 .

[28]  Harold L. Somers,et al.  Round-trip Translation: What Is It Good For? , 2005, ALTA.

[29]  Sergio Segura.  Metamorphic Testing: Challenges Ahead , 2018, 2018 IEEE/ACM 3rd International Workshop on Metamorphic Testing (MET).

[30]  Sergio Segura,et al.  Metamorphic Testing: Testing the Untestable , 2020, IEEE Software.