AAAI Publications, Second AAAI Conference on Human Computation and Crowdsourcing

Font Size: 
Wish: Amplifying Creative Ability with Expert Crowds
Anand Kulkarni, Prayag Narula, David Rolnitzky, Nathan Kontny

Last modified: 2014-09-05


We present Wish, a system that uses expert members of an online crowd to amplify a user’s ability to carry out complex creative tasks.  When presented with an incomplete or rough draft of creative content (for example, writing, programming, or art), Wish finds and recruits an expert to suggest a possible realization of the user’s vision. Wish contributes a new approach to crowdsourcing complex work. Rather than combining pools of unskilled workers to producing complex output, Wish uses the crowd to identify and recruit a pool of experts, then assigns a single high-quality expert to carry out the task.  This approach retains the benefits of using a crowd — scalability, speed, correctness, and responsiveness — while simplifying the process of crowdsourcing complex work. We demonstrate Wish in the context of three prototype tools.  Draft enables users to consult a crowd of authors for suggestions on writing.  Hack lets developers convert pseudocode into working code by pulling results on demand from a crowd of programmers. Sketch lets individuals convert rough sketches into fully-refined art by consulting artists.  We illustrate how novice creators can use Wish to amplify their ability, how expert designers can use Wish to explore design spaces and improve the speed of creation, and how new users can use Wish to gain feedback from experts.


1. Licklider, J.C.R., "Man-Computer Symbiosis", IRE Transactions on Human Factors in Electronics, vol. HFE-1, 4-11, Mar 1960.

2. Bush, Vannevar. "As We May Think." The Atlantic. July 1945.

3. Von Ahn, L.; Dabbish, L. (2004). "Labeling images with a computer game".Proceedings of the 2004 conference on Human factors in computing systems - CHI '04.

4. Little, G., Chilton, L. B., Goldman, M., & Miller, R. C. (2010, July). Exploring iterative and parallel human computation processes. In Proceedings of the ACM SIGKDD workshop on human computation (pp. 68-76). ACM.

5. oDesk.

6. Retelny, D., Robaszkiewicz, S., To, A., & Bernstein, M. Enabling Expert Crowdsourcing with Flash Teams. Crowdconf 2013.

7. Zhang, H., Lai, J. K., & Bächer, M. (2012, July). Hallucination: A mixed-initiative approach for efficient document reconstruction. In Workshops at the Twenty-Sixth AAAI Conference on Artificial Intelligence.

8. Bigham, J. P., Jayant, C., Ji, H., Little, G., Miller, A., Miller, R. C., & Yeh, T. (2010, October). Vizwiz: nearly real-time answers to visual questions. In Proceedings of the 23nd annual ACM symposium on User interface software and technology (pp. 333-342). ACM.

9. Bernstein, M. S., Brandt, J., Miller, R. C., & Karger, D. R. (2011, October). Crowds in two seconds: Enabling realtime crowd-powered interfaces. In Proceedings of the 24th annual ACM symposium on User interface software and technology (pp. 33-42). ACM.

10. Dow, S. P., Fortuna, J., Schwartz, D., Altringer, B., Schwartz, D. L., & Klemmer, S. R. (2012). Prototyping dynamics: sharing multiple designs improves exploration, group rapport, and results. In Design Thinking Research(pp. 47-70). Springer Berlin Heidelberg.

11. Dow, S., Gerber, E., & Wong, A. (2013, April). A pilot study of using crowds in the classroom. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 227-236). ACM.

12. LaToza, T. D., Towne, W. B., van der Hoek, A., & Herbsleb, J. D. (2013, May). Crowd Development. In Cooperative and Human Aspects of Software Engineering (CHASE), 2013 6th International Workshop on (pp. 85-88). IEEE.

13. Little, Greg. 2012. HumanScript.

15. Kleinberg, J., & Raghavan, P. (2005, October). Query incentive networks. In Foundations of Computer Science, 2005. FOCS 2005. 46th Annual IEEE Symposium on (pp. 132-141). IEEE.

15. Pickard, G., Pan, W., Rahwan, I., Cebrian, M., Crane, R., Madan, A., & Pentland, A. (2011). Time-critical social mobilization. Science334(6055), 509-512.

16. Kulkarni, A., Gutheim, P., Narula, P., Rolnitzky, D., Parikh, T., & Hartmann, B. (2012). Mobileworks: Designing for quality in a managed crowdsourcing architecture. Internet Computing, IEEE16(5), 28-35.

17. Dow, S., Kulkarni, A., Klemmer, S., & Hartmann, B. (2012, February). Shepherding the crowd yields better work. In Proceedings of the ACM 2012 conference on Computer Supported Cooperative Work (pp. 1013-1022). ACM.

18. Kulkarni, A., Can, M., & Hartmann, B. (2012, February). Collaboratively crowdsourcing workflows with Turkomatic. In Proceedings of the ACM 2012 conference on Computer Supported Cooperative Work (pp. 1003-1012). ACM.

Full Text: PDF