Understanding Crowdsourcing Workflow: Modeling and Optimizing Iterative and Parallel Processes

2016-11-03

Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, 4

Volume

Vol. 4 (2016): Fourth AAAI Conference on Human Computation and Crowdsourcing

Full Papers

The advantages and disadvantages of different crowdsourcing workflow structures have been analyzed. Existing studies on crowdsourcing workflow mainly focused on the quality control of the tasks using iterative and parallel processes. On the other hand, the characteristics of workflow considering the various task and crowdsourcing environments are not yet fully analyzed. Therefore, we face two difficulties in making use of workflow: the workflow optimization and the prior quality estimation. This research proposes the crowdsourcing workflow model for the set of improvement tasks, considering the ability distribution of crowdsourcing workers, improvement difficulty of the task, and the preference of the requester. In addition, we show the optimal workflow can be found by the search algorithm on the proposed model. The result of optimization can be used for both constructing the best workflow and estimating the quality of task performance. Experimental results under various conditions indicate that the degree of parallelism of the optimal workflow increases with the variance of worker ability. Also, iterative processes should be used when the average ability of the workers trends away from the middle level. These results include the existing research and therefore the model presented is useful in understanding crowdsourcing workflows.

10.1609/hcomp.v4i1.13289