Utilizing Crowdsourced Asynchronous Chat for Efficient Collection of Dialogue Dataset

2018-07-09

Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, 6

Volume

Vol. 6 (2018): Sixth AAAI Conference on Human Computation and Crowdsourcing

Technical Papers

In this paper, we design a crowd-powered system to efficiently collect data for training dialogue systems. Conventional systems assign dialogue roles to a pair of crowd workers, and record their interaction on an online chat. In this framework, the pair is required to work simultaneously, and one worker must wait for the other when he/she is writing a message, which decreases work efficiency. Our proposed system allows multiple workers to create dialogues in an asynchronous manner, which relieves workers from time restrictions. We have conducted an experiment using our system on a crowdsourcing platform to evaluate the efficiency and the quality of dialogue collection. Results show that our system can reduce the necessary time to input a message by 68% while maintaining quality.

10.1609/hcomp.v6i1.13321