Stewart, N., Chandler, J., & Paolacci, J. (2017). Crowdsourcing cognitive science. Trends in Cognitive Sciences, 21, 736-748. doi: 10.1016/j.tics.2017.06.007

Crowdsourcing data collection with research participants from online labor markets is now well established in cognitive science. We review who is in the crowd and who can be reached by the average laboratory. We discuss reproducibility and review some recent methodological innovations for online experiments. We consider the design of research studies and arising ethical issues. We review how to code experiments for the Web, and what is known about video and audio presentation and the measurement of reaction times. We close with comments about the high levels of experience of many participants and an arising tragedy of the commons.