AAAI Publications, First AAAI Conference on Human Computation and Crowdsourcing

Font Size: 
Crowdsourcing Quality Control for Item Ordering Tasks
Toshiko Matsui, Yukino Baba, Toshihiro Kamishima, Hisashi Kashima

Last modified: 2013-11-03


One of the biggest challenges in crowdsourcing is quality control which is to expect high quality results from crowd workers who are not necessarily very capable nor motivated.In this paper, we consider item ordering questions, where workersare asked to arrange multiple items in the correct order. We propose a probabilistic generative model of crowd answers by extending a distance-based order model to incorporate worker ability, and give an efficient estimation algorithm.

Full Text: PDF