A taxonomy of quality assessment methods for volunteered and crowdsourced geographic information

Abstract The growing use of crowdsourced geographic information (CGI) has prompted the employment of several methods for assessing information quality, which are aimed at addressing concerns on the lack of quality of the information provided by non‐experts. In this work, we propose a taxonomy of methods for assessing the quality of CGI when no reference data are available, which is likely to be the most common situation in practice. Our taxonomy includes 11 quality assessment methods that were identified by means of a systematic literature review. These methods are described in detail, including their main characteristics and limitations. This taxonomy not only provides a systematic and comprehensive account of the existing set of methods for CGI quality assessment, but also enables researchers working on the quality of CGI in various sources (e.g., social media, crowd sensing, collaborative mapping) to learn from each other, thus opening up avenues for future work that combines and extends existing methods into new application areas and domains.

[1]  Ran R. Hassin,et al.  Can Consumers Make Affordable Care Affordable? The Value of Choice Architecture , 2013, PloS one.

[2]  Werner Kuhn,et al.  Trust and Reputation Models for Quality Assessment of Human Sensor Observations , 2013, COSIT.

[3]  Virach Sornlertlamvanich,et al.  Reliable Improvement for Collective Intelligence on Thai Herbal Information , 2010, Advances in Intelligent Information and Database Systems.

[4]  Carl F. Salk,et al.  Comparing the Quality of Crowdsourced Data Contributed by Expert and Non-Experts , 2013, PloS one.

[5]  Alexander Zipf,et al.  A Conceptual Quality Framework for Volunteered Geographic Information , 2015, COSIT.

[6]  Yue Cui,et al.  A systematic approach to evaluate and validate the spatial accuracy of farmers market locations using multi-geocoding services , 2013 .

[7]  Gloria Bordogna,et al.  On predicting and improving the quality of Volunteer Geographic Information projects , 2016, Int. J. Digit. Earth.

[8]  Steffen Fritz,et al.  Limitations of Majority Agreement in Crowdsourced Image Interpretation , 2017, Trans. GIS.

[9]  Bernd Resch,et al.  People as Sensors and Collective Sensing-Contextual Observations Complementing Geo-Sensor Network Measurements , 2013, Progress in Location-Based Services.

[10]  Andrea Cavallaro,et al.  Sensor Capability and Atmospheric Correction in Ocean Colour Remote Sensing , 2015, Remote. Sens..

[11]  Pearl Brereton,et al.  Performing systematic literature reviews in software engineering , 2006, ICSE.

[12]  Steffen Fritz,et al.  Assessing quality of volunteer crowdsourcing contributions: lessons from the Cropland Capture game , 2016, Int. J. Digit. Earth.

[13]  Paula Gomes Mian,et al.  Systematic Review in Software Engineering , 2005 .

[14]  Christoph Perger,et al.  Using control data to determine the reliability of volunteered geographic information about land cover , 2013, Int. J. Appl. Earth Obs. Geoinformation.

[15]  Carsten Keßler,et al.  Trust as a Proxy Measure for the Quality of Volunteered Geographic Information in the Case of OpenStreetMap , 2013, AGILE Conf..

[16]  Melanie Eckle,et al.  The Tasks of the Crowd: A Typology of Tasks in Geographic Information Crowdsourcing and a Case Study in Humanitarian Mapping , 2016, Remote. Sens..

[17]  Vyron Antoniou,et al.  How Many Volunteers Does it Take to Map an Area Well? The Validity of Linus’ Law to Volunteered Geographic Information , 2010 .

[18]  Dennis Zielstra,et al.  Positional accuracy analysis of Flickr and Panoramio images for selected world regions , 2013 .

[19]  Stefan Stieglitz,et al.  Volunteered Geographic Information and Its Implications for Disaster Management , 2016, 2016 49th Hawaii International Conference on System Sciences (HICSS).

[20]  Pascal Neis,et al.  A Comprehensive Framework for Intrinsic OpenStreetMap Quality Analysis , 2014, Trans. GIS.

[21]  M. Goodchild Citizens as sensors: the world of volunteered geography , 2007 .

[22]  A. Stefanidis,et al.  Harvesting ambient geospatial information from social media feeds , 2011, GeoJournal.

[23]  Sven G. Bilen,et al.  Increasing the veracity of event detection on social media networks through user trust modeling , 2014, 2014 IEEE International Conference on Big Data (Big Data).

[24]  Hansi Senaratne,et al.  A review of volunteered geographic information quality assessment methods , 2017, Int. J. Geogr. Inf. Sci..

[25]  Miriam J. Metzger,et al.  The credibility of volunteered geographic information , 2008 .

[26]  Guillaume Touya,et al.  Quality Assessment of the French OpenStreetMap Dataset , 2010, Trans. GIS.

[27]  Michele Melchiori,et al.  A Crowdsourcing-Based Framework for Improving Geo-spatial Open Data , 2013, 2013 IEEE International Conference on Systems, Man, and Cybernetics.

[28]  Giles M. Foody,et al.  Rating crowdsourced annotations: evaluating contributions of variable quality and completeness , 2014, Int. J. Digit. Earth.

[29]  Jan Muntermann,et al.  A method for taxonomy development and its application in information systems , 2013, Eur. J. Inf. Syst..

[30]  João Porto de Albuquerque,et al.  Flood Citizen Observatory: a crowdsourcing-based approach for flood risk management in Brazil , 2014, SEKE.

[31]  João Porto de Albuquerque,et al.  Crowdsourced Validation and Updating of Dynamic Features in OpenStreetMap - An analysis of Shelter Mapping after the 2015 Nepal Earthquake , 2016, ISCRAM.

[32]  A. Kaplan,et al.  Users of the world, unite! The challenges and opportunities of Social Media , 2010 .

[33]  Adam C. Winstanley,et al.  Towards quality metrics for OpenStreetMap , 2010, GIS '10.

[34]  Alexander Zipf,et al.  A geographic approach for combining social media and authoritative data towards identifying useful information for disaster management , 2015, Int. J. Geogr. Inf. Sci..

[35]  Steffen Fritz,et al.  Assessing the Accuracy of Volunteered Geographic Information arising from Multiple Contributors to an Internet Based Collaborative Project , 2013, Trans. GIS.

[36]  K. Janowicz,et al.  Can we Trust Information ?-The Case of Volunteered Geographic Information , 2010 .

[37]  Ahmed Loai Ali,et al.  Data Quality Assurance for Volunteered Geographic Information , 2014, GIScience.

[38]  Bernhard Höfle,et al.  Quality Improvement of Remotely Volunteered Geographic Information via Country-Specific Mapping Instructions , 2017, ISCRAM.

[39]  Siegfried Reich,et al.  What is an Appropriate Temporal Sampling Rate to Record Floating Car Data with a GPS? , 2016, ISPRS Int. J. Geo Inf..

[40]  Marco Minghini,et al.  A Review of OpenStreetMap Data , 2017 .

[41]  Vyron Antoniou,et al.  MEASURES AND INDICATORS OF VGI QUALITY: AN OVERVIEW , 2015 .

[42]  Gloria Bordogna,et al.  A linguistic decision making approach to assess the quality of volunteer geographic information for citizen science , 2014, Inf. Sci..

[43]  Pascal Neis,et al.  Quality assessment for building footprints data on OpenStreetMap , 2014, Int. J. Geogr. Inf. Sci..

[44]  Michael F. Goodchild,et al.  Assuring the quality of volunteered geographic information , 2012 .

[45]  De Groeve Tom,et al.  Citizens as Sensors for Natural Hazards: A VGI integration Workflow , 2010 .

[46]  Alexander Zipf,et al.  Crowdsourcing geographic information for disaster management and improving urban resilience: an overview of recent developments and lessons learned , 2016 .

[47]  Francis Harvey,et al.  To Volunteer or to Contribute Locational Information? Towards Truth in Labeling for Crowdsourced Geographic Information , 2013 .

[48]  Kevin Crowston,et al.  Mechanisms for Data Quality and Validation in Citizen Science , 2011, 2011 IEEE Seventh International Conference on e-Science Workshops.

[49]  Bin Jiang,et al.  Crowdsourcing, Citizen Science or Volunteered Geographic Information? The Current State of Crowdsourced Geographic Information , 2016, ISPRS Int. J. Geo Inf..

[50]  Alexander Zipf,et al.  A Framework of Quality Assessment Methods for Crowdsourced Geographic Information: a Systematic Literature Review , 2017, ISCRAM.

[51]  Pearl Brereton,et al.  Lessons from applying the systematic literature review process within the software engineering domain , 2007, J. Syst. Softw..

[52]  W. Tobler A Computer Movie Simulating Urban Growth in the Detroit Region , 1970 .

[53]  MohammadReza Malek,et al.  Artificial intelligence-based solution to estimate the spatial accuracy of volunteered geographic data , 2015 .

[54]  Hansi Senaratne,et al.  Using Reverse Viewshed Analysis to Assess the Location Correctness of Visually Generated VGI , 2013, Trans. GIS.

[55]  Michela Bertolotto,et al.  Automated highway tag assessment of OpenStreetMap road networks , 2014, SIGSPATIAL/GIS.

[56]  John Krumm,et al.  Discovering points of interest from users’ map annotations , 2008 .

[57]  Javier Nogueras-Iso,et al.  A method for checking the quality of geographic metadata based on ISO 19157 , 2018, Int. J. Geogr. Inf. Sci..