Assessing quality of volunteer crowdsourcing contributions: lessons from the Cropland Capture game
- 23 March 2015
- journal article
- research article
- Published by Taylor & Francis Ltd in International Journal of Digital Earth
- Vol. 9 (4), 410-426
- https://doi.org/10.1080/17538947.2015.1039609
Abstract
Volunteered geographic information (VGI) is the assembly of spatial information based on public input. While VGI has proliferated in recent years, assessing the quality of volunteer-contributed data has proven challenging, leading some to question the efficiency of such programs. In this paper, we compare several quality metrics for individual volunteers’ contributions. The data were the product of the ‘Cropland Capture’ game, in which several thousand volunteers assessed 165,000 images for the presence of cropland over the course of 6 months. We compared agreement between volunteer ratings and an image's majority classification with volunteer self-agreement on repeated images and expert evaluations. We also examined the impact of experience and learning on performance. Volunteer self-agreement was nearly always higher than agreement with majority classifications, and much greater than agreement with expert validations although these metrics were all positively correlated. Volunteer quality showed a broad trend toward improvement with experience, but the highest accuracies were achieved by a handful of moderately active contributors, not the most active volunteers. Our results emphasize the importance of a universal set of expert-validated tasks as a gold standard for evaluating VGI quality.Keywords
Funding Information
- European Research Council (CrowdLand (617754),SIGMA (603719))
- International Institute for Applied Systems Analysis (Postdoctoral Fellowship)
This publication has 27 references indexed in Scilit:
- Measuring Completeness of Building Footprints in OpenStreetMap over Space and TimeISPRS International Journal of Geo-Information, 2013
- The Rise of Collaborative Mapping: Trends and Future DirectionsISPRS International Journal of Geo-Information, 2013
- A Photogrammetric Approach for Assessing Positional Accuracy of OpenStreetMap© RoadsISPRS International Journal of Geo-Information, 2013
- The Street Network Evolution of Crowdsourced Maps: OpenStreetMap in Germany 2007–2011Future Internet, 2011
- How Good is Volunteered Geographical Information? A Comparative Study of OpenStreetMap and Ordnance Survey DatasetsEnvironment and Planning B: Planning and Design, 2010
- Quality Assessment of the French OpenStreetMap DatasetTransactions in GIS, 2010
- MODIS Collection 5 global land cover: Algorithm refinements and characterization of new datasetsRemote Sensing of Environment, 2010
- Geo-Wiki.Org: The Use of Crowdsourcing to Improve Global Land CoverRemote Sensing, 2009
- Goodness of Fit and Related Inference Processes for Quantile RegressionJournal of the American Statistical Association, 1999
- Maximum Likelihood Estimation of Observer Error-Rates Using the EM AlgorithmJournal of the Royal Statistical Society Series C: Applied Statistics, 1979