Skip to main content
Research article - Peer-reviewed, 2016

Assessing quality of volunteer crowdsourcing contributions: lessons from the Cropland Capture game

Salk, Carl; Sturn, Tobias; See, Linda; Fritz, Steffen; Perger, Christoph

Abstract

Volunteered geographic information (VGI) is the assembly of spatial information based on public input. While VGI has proliferated in recent years, assessing the quality of volunteer-contributed data has proven challenging, leading some to question the efficiency of such programs. In this paper, we compare several quality metrics for individual volunteers' contributions. The data were the product of the Cropland Capture' game, in which several thousand volunteers assessed 165,000 images for the presence of cropland over the course of 6 months. We compared agreement between volunteer ratings and an image's majority classification with volunteer self-agreement on repeated images and expert evaluations. We also examined the impact of experience and learning on performance. Volunteer self-agreement was nearly always higher than agreement with majority classifications, and much greater than agreement with expert validations although these metrics were all positively correlated. Volunteer quality showed a broad trend toward improvement with experience, but the highest accuracies were achieved by a handful of moderately active contributors, not the most active volunteers. Our results emphasize the importance of a universal set of expert-validated tasks as a gold standard for evaluating VGI quality.

Keywords

crowdsourcing; volunteered geographic information; cropland; data quality; image classification; Geo-Wiki

Published in

International Journal of Digital Earth
2016, volume: 9, number: 4, pages: 410-426
Publisher: TAYLOR & FRANCIS LTD

Authors' information

International Institute for Applied Systems Analysis (IIASA)
Swedish University of Agricultural Sciences, Southern Swedish Forest Research Centre
Sturn, Tobias
International Institute for Applied Systems Analysis (IIASA)
See, Linda
International Institute for Applied Systems Analysis (IIASA)
Fritz, Steffen
International Institute for Applied Systems Analysis (IIASA)
Perger, Christoph
International Institute for Applied Systems Analysis (IIASA)

UKÄ Subject classification

Geosciences, Multidisciplinary
Environmental Sciences

Publication Identifiers

DOI: https://doi.org/10.1080/17538947.2015.1039609

URI (permanent link to this page)

https://res.slu.se/id/publ/79985