AAAI Publications, Second AAAI Conference on Human Computation and Crowdsourcing

Font Size: 
Incentives to Counter Bias in Human Computation
Boi Faltings, Radu Jurca, Pearl Pu, Bao Duy Tran

Last modified: 2014-09-05


In online labor platforms such as Amazon Mechanical Turk, a good strategy to obtain quality answers is to take aggregate answers submitted by multiple workers, exploiting the wisdom of crowds. However, human computation issusceptible to systematic biases which cannot be corrected by using multiple workers.We investigate a game-theoretic bonus scheme, called peer truth serum (PTS), to overcome this problem. We report on the design and outcomes of a set of experiments to validate this scheme. Results show  peer truth serum can indeed correct the biases and increase the answer accuracy by up to 80%.


L. von Ahn and L. Dabbish. Labeling Images with a Computer

Game. ACM Conference on Human Factors in Computing

Systems , pp. 319-326, 2004.

A. Dasgupta and A. Ghosh: Crowdsourced Judgement Elicitation

with Endogenous Proficiency. WWW Conference,


B. Faltings, J.J. Li and R. Jurca. Incentive Mechanisms for

Community Sensing. IEEE Transactions on Computers ,

63 (1), pp. 115-128, 2014.

X. A. Gao, A. Mao, Y. Chen and R. P. Adam. Trick or Treat:

Putting Peer Prediction to the Test. Proc. of the 15th ACM

Conference on Economics and Computation (EC), 2014.

F. Garcin and B. Faltings. Swissnoise: Online Polls with

Game-Theoretic Incentives. Proceedings of the 26th IAAI,


C. G. Harris. You’re Hired! An Examination of Crowdsourcing

Incentive Models in Human Resource Tasks. WSDM

2011 Workshop on Crowdsourcing for Search and Data

Mining, pp. 15-18, 2011.

J. J. Horton, D.G. Rand and R. Zeckhauser. The Online

Laboratory: Conducting Experiments in a Real Labor Market.

Faculty research paper RWP10-017, Harvard Kennedy

School, 2010.

S.-W. Huang and W.-T. Fu. Enhancing Reliability Using

Peer Consistency Evaluation in Human Computation. Proceedings

of the 2013 conference on Computer supported cooperative

work , pp. 639-648, 2013

S.-W. Huang and W.-T. Fu. Don’t Hide in the Crowd! Increasing

Social Transparency Between Peer Workers Improves

Crowdsourcing Outcomes. CHI , pp. 621-630, 2013

P. Ipeirotis. Demographics of Mechanical Turk. New York

University Working Paper No. CEDER-10-01, 2010.

P. Ipeirotis, F.Provost and J. Wang. Quality Management

on amazon mechanical turk.ACM SIGKDD Workshop

HCOMP’10 , pp. 64-67, 2010.

J. Lorenz, H. Rauhut, F. Schweitzer and D. Helbing. How

social influence can undermine the wisdom of crowd effect.

Proceedings of the National Academy of Sciences, 108, pp.

9020-9025, 2011.

R. Jurca and B. Faltings. An Incentive Compatible Reputation

Mechanism. Proceedings of the IEEE Conference on

E-Commerce, pp. 285-292, 2003.

R. Jurca and B. Faltings. Mechanisms for Making Crowds

Truthful. Journal of Artificial Intelligence Research (JAIR) ,

34, 2009, pp. 209-253.

R. Jurca and B. Faltings. Incentives for Answering Hypothetical

Questions. Workshop on Social Computing and User

Generated Content , ACM Conference on Electronic Commerce,

San Jose, 2011.

E. Kamar and E. Horvitz. Incentives for Truthful Reporting

in Crowdsourcing. Proceedings of AAMAS 2012 , Valencia,


N. Lambert and Y. Shoham. Eliciting Truthful Answers

to Multiple-Choice Questions. In Proceedings of the tenth

ACM conference on Electronic Commerce , pp. 109-118,


N. Miller, P. Resnick, and R. Zeckhauser. Eliciting Informative

Feedback: The Peer-Prediction Method. Management

Science , 51:1359–1373, 2005.

A. Papakonstantinou, A. Rogers, E. H. Gerding, and N. R.

Jennings. Mechanism design for the truthful elicitation of

costly probabilistic estimates in distributed information systems.

Artif. Intell. , 175(2):648–672, 2011.

D. Prelec. A Bayesian Truth Serum for Subjective Data. Science ,

306(5695), pp. 462-466, 2004.

G. Radanovic and B. Faltings. A Robust Bayesian Truth

Serum for Non-binary Signals. Proceedings of the 27th

AAAI Conference on Artificial Intelligence (AAAI’13) , pages

833-839, 2013

G. Radanovic and B. Faltings. Incentives for Truthful Information

Elicitation of Continuous Signals. Proceedings of the

28th AAAI Conference on Artificial Intelligence (AAAI’14) ,


G. Radanovic and B. Faltings. Incentives for Effort in

Crowdsourcing using the Peer Truth Serum. submitted for

publication , 2014.

L. J. Savage. Elicitation of Personal Probabilities and Expectations.

Journal of the American Statistical Association ,

66(336):783–801, 1971.

A. Shaw, J. J. Horton and D. L. Chen. Designing Incentives

for Inexpert Human Raters. Proceedings of the ACM Conference

on Computer Supported Cooperative Work, 2010

A. Tversky and D. Kahneman. Judgment under uncertainty:

Heuristics and biases. Science , 185, pp. 11241130, 1974

J. Witkowski and D. C. Parkes. Peer Prediction without a

Common Prior. Proceedings of the 13th ACM Conference

on Electronic Commerce (EC 2012) , 2012.

J. Witkowski and D. C. Parkes. A Robust Bayesian Truth

Serum for Small Populations. Proceedings of the 26th AAAI

Conference on Artificial Intelligence (AAAI 2012) , 2012.

A. Zohar and J. S. Rosenschein. Robust Mechanisms for

Information Elicitation, in AAAI ’06: Proceedings of The

Twenty-First National Conference on Artificial Intelligence,

July 16-20, AAAI Press, Menlo Park, CA, USA, 2006.

Full Text: PDF