What Will Others Choose? How a Majority Vote Reward Scheme Can Improve Human Computation in a Spatial Location Identification Task

2013-11-10

Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, 1

Volume

Vol. 1 (2013): First AAAI Conference on Human Computation and Crowdsourcing

Full Papers

We created a spatial location identification task (SpLIT) in which workers recruited from Amazon Mechanical Turk were presented with a camera view of a location, and were asked to identify the location on a two-dimensional map. In cases where these cues were ambiguous or did not provide enough information to pinpoint the exact location, workers had to make a best guess. We tested the effects of two reward schemes. In the “ground truth” scheme, workers were rewarded if their answers were close enough to the correct locations. In the “majority vote” scheme, workers were told that they would be rewarded if their answers were similar to the majority of other workers. Results showed that the majority vote reward scheme led to consistently more accurate answers. Cluster analysis further showed that the majority vote reward scheme led to answers with higher reliability (a higher percentage of answers in the correct clusters) and precision (a smaller average distance to the cluster centers). Possible reasons for why the majority voting reward scheme was better were discussed.

10.1609/hcomp.v1i1.13082