WingIt: Efficient Refinement of Unclear Task Instructions

2018-07-09

Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, 6

Volume

Vol. 6 (2018): Sixth AAAI Conference on Human Computation and Crowdsourcing

Technical Papers

Crowdsourcing has the inherent potential to shift tedious work from requesters with limited time (or patience) to an on-demand workforce. However, the time saved by requesters is offset by the time they must spend preparing instructions and refining them to address the ambiguities that typically arise. Hastily written instructions should be welcomed and supported. This paper presents a set of methods that enable workers to cope with unclear or ambiguous instructions and produce high quality results with minimal reliance on the requester. Workers’ intuition about the requester’s needs is leveraged to move onus for answering questions and revising instructions to workers. Our system, WingIt, implements these methods and demonstrates the relative tradeoffs between resolving by questions versus editing instructions directly, and between waiting for immediate response from the requester versus allowing workers to proceed based on their best guess of the requester’s intent.

10.1609/hcomp.v6i1.13338