AAAI Publications, Fifth AAAI Conference on Human Computation and Crowdsourcing

Font Size: 
Deja Vu: Characterizing Worker Reliability Using Task Consistency
Alex C. Williams, Joslin Goh, Charlie G. Willis, Aaron M. Ellison, James H. Brusuelas, Charles C. Davis, Edith Law

Last modified: 2017-09-21


Consistency is a practical metric that evaluates an instrument's reliability based on its ability to yield the same output when repeatedly given a particular input. Despite its broad usage, little is understood about the feasibility of using consistency as a measure of worker reliability in crowdwork. In this paper, we explore the viability of measuring a worker's reliability by their ability to conform to themselves. We introduce and describe Deja Vu, a mechanism for dynamically generating task queues with consistency probes to measure the consistency of workers who repeat the same task twice. We present a study that utilizes Deja Vu to examine how generic characteristics of the duplicate task - such as placement, difficulty, and transformation - affect a worker’s task consistency in the context of two unique object detection tasks. Our findings provide insight into the design and use of consistency-based reliability metrics.


consistency; reliability; crowdsourcing

Full Text: PDF