Towards evaluating crowdsourced image classification on mobile devices to generate geographic information about human settlements

Geographic information crowdsourcing is an increasingly popular approach to derive geographic data about human settlements from remotely sensed imagery. However, crowdsourcing approaches are frequently associated with uncertainty about the quality of the information produced. Although previous studies have found acceptable quality of crowdsourced information in some application domains, there is still lack of research about the quality of information produced with mobile crowdsourcing tools. This paper aims to contribute towards filling this gap by presenting an initial analysis of the contributions of two crowdsourcing projects based on the MapSwipe mobile app, in Madagascar and South Sudan. Our results show, that there is substantial agreement amongst volunteers thus suggesting that mobile crowdsourcing is a viable approach to support the mapping of human settlements. Nevertheless, this study also identifies several factors that may cause disagreement between volunteers (e.g. bad imagery, dependence on individual users) and thus reduce the reliability of the information they produce.

[1]  Steffen Fritz,et al.  Geo-Wiki.Org: The Use of Crowdsourcing to Improve Global Land Cover , 2009, Remote. Sens..

[2]  Melanie Eckle,et al.  The Tasks of the Crowd: A Typology of Tasks in Geographic Information Crowdsourcing and a Case Study in Humanitarian Mapping , 2016, Remote. Sens..

[3]  J. R. Landis,et al.  The measurement of observer agreement for categorical data. , 1977, Biometrics.

[4]  P. Moran.  Notes on continuous stochastic phenomena. , 1950, Biometrika.

[5]  Jonathan Corpus Ong,et al.  Tweeting Supertyphoon Haiyan: Evolving Functions of Twitter during and after a Disaster Event , 2016, PloS one.

[6]  Norman Kerle,et al.  Collaborative damage mapping for emergency response: the role of Cognitive Systems Engineering , 2013 .

[7]  J. Sim,et al.  The kappa statistic in reliability studies: use, interpretation, and sample size requirements. , 2005, Physical therapy.

[8]  Clay Westrope,et al.  Groundtruthing OpenStreetMap building damage assessment , 2014 .

[9]  Fernando M. Ramos,et al.  Methods for evaluating volunteers' contributions in a deforestation detection citizen science project , 2016, Future Gener. Comput. Syst..

[10]  Fernando Diaz,et al.  Extracting information nuggets from disaster- Related messages in social media , 2013, ISCRAM.

[11]  J. Fleiss.  Measuring nominal scale agreement among many raters. , 1971 .

[12]  A. Feinstein,et al.  High agreement but low kappa: I. The problems of two paradoxes. , 1990, Journal of clinical epidemiology.

[13]  Steffen Fritz,et al.  Development of a global hybrid forest mask through the synergy of remote sensing, crowdsourcing and FAO statistics , 2015 .

[14]  John P. Wilson,et al.  Conducting disaster damage assessments with Spatial Video, experts, and citizens , 2014 .