AAAI Publications, First AAAI Conference on Human Computation and Crowdsourcing

Font Size: 
A Human-Centered Framework for Ensuring Reliability on Crowdsourced Labeling Tasks
Omar Alonso, Catherine C. Marshall, Marc A. Najork

Last modified: 2013-11-03


This paper describes an approach to improving the reliability of a crowdsourced labeling task for which there is no objective right answer. Our approach focuses on three contingent elements of the labeling task: data quality, worker reliability, and task design. We describe how we developed and applied this framework to the task of labeling tweets according to their interestingness. We use in-task CAPTCHAs to identify unreliable workers, and measure inter-rater agreement to decide whether subtasks have objective or merely subjective answers.


crowdsourcing;label quality;experimental design; CAPTCHA

Full Text: PDF