Metamorphic Testing of Mapping Software

Mapping software is difficult to test because it is very costly to evaluate its output. This difficulty is generally known as the oracle problem, a fundamental challenge in software testing. In this paper, we propose a metamorphic testing strategy to alleviate the oracle problem in testing mapping software. We first conduct a case study to test Google Maps, the most popular web mapping service. The results of the case study show that our testing approach is effective, with the detection of several real-life bugs that can hardly be exposed under conventional testing paradigms. Following this, we conduct an analysis of the system OpenStreetMap, well-known open-source mapping software built and maintained by a community of users. We show the potential of metamorphic testing for such systems. These case studies show that metamorphic testing can be applied to mapping software for both verification and validation purposes.

[1]  Tsong Yueh Chen,et al.  Fault-based testing without the need of oracles , 2003, Inf. Softw. Technol..

[2]  Sergio Segura,et al.  [Journal First] Metamorphic Testing of RESTful Web APIs , 2018, 2018 IEEE/ACM 40th International Conference on Software Engineering (ICSE).

[3]  Caroline Geck,et al.  The World Factbook , 2017 .

[4]  Tsong Yueh Chen,et al.  Metamorphic Testing for Software Quality Assessment: A Study of Search Engines , 2016, IEEE Transactions on Software Engineering.

[5]  John L. Goodman,et al.  The Space Shuttle and GPS: A Safety-Critical Navigation Upgrade , 2003, ICCBSS.

[6]  J. L. Goodman,et al.  A Software Perspective on GNSS Receiver Integration and Operation , 2003 .

[7]  Adel M. Alimi,et al.  Mining road map from big database of GPS data , 2014, 2014 14th International Conference on Hybrid Intelligent Systems.

[8]  Dave Towey,et al.  Metamorphic Relations for Enhancing System Understanding and Use , 2020, IEEE Transactions on Software Engineering.

[9]  Tsong Yueh Chen,et al.  Metamorphic testing for Adobe data analytics software , 2017 .

[10]  Mikael Lindvall,et al.  Metamorphic Model-Based Testing Applied on NASA DAT -- An Experience Report , 2015, 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering.

[11]  Jeffrey M. Voas,et al.  Metamorphic Testing for Cybersecurity , 2016, Computer.

[12]  Tsong Yueh Chen,et al.  Metamorphic Testing: A New Approach for Generating Next Test Cases , 2020, ArXiv.

[13]  R. Johannessen,et al.  TEN MILLION DATA POINTS FROM TSO-APPROVED AVIATION GPS RECEIVERS: RESULTS OF ANALYSIS AND APPLICATIONS TO DESIGN AND USE IN AVIATION , 2000 .

[14]  Tsong Yueh Chen,et al.  An effective testing method for end-user programmers , 2005, ACM SIGSOFT Softw. Eng. Notes.

[15]  Huai Liu,et al.  How Effectively Does Metamorphic Testing Alleviate the Oracle Problem? , 2014, IEEE Transactions on Software Engineering.

[16]  Sergio Segura,et al.  A Survey on Metamorphic Testing , 2016, IEEE Transactions on Software Engineering.

[17]  Mark Harman,et al.  The Oracle Problem in Software Testing: A Survey , 2015, IEEE Transactions on Software Engineering.

[18]  P. Mooney,et al.  Comparison of the accuracy of OpenStreetMap for Ireland with Google Maps and Bing Maps , 2010 .

[19]  Hans-Peter Kriegel,et al.  MARiO: Multi-Attribute Routing in Open Street Map , 2011, SSTD.

[20]  Tsong Yueh Chen,et al.  Case studies on the selection of useful relations in metamorphic testing , 2004 .

[21]  Liqun Sun,et al.  Metamorphic testing of driverless cars , 2019, Commun. ACM.

[22]  Yang-Wai Chow,et al.  Metamorphic Testing of Navigation Software: A Pilot Study with Google Maps , 2018, HICSS.

[23]  Michael P. Wright,et al.  The effectiveness of global positioning system electronic navigation , 2003, IEEE SoutheastCon, 2003. Proceedings..

[24]  Xin-Hua Hu,et al.  A Machine Learning Based Framework for Verification and Validation of Massive Scale Image Data , 2021, IEEE Transactions on Big Data.

[25]  Huai Liu,et al.  Metamorphic Testing , 2018, ACM Comput. Surv..

[26]  A. Alimi,et al.  Collection and exploration of GPS based vehicle traces database , 2015, 2015 4th International Conference on Advanced Logistics and Transport (ICALT).

[27]  Dennis Luxen,et al.  Real-time routing with OpenStreetMap data , 2011, GIS.