Researchers who study interventions for speech disorders need to obtain blinded listeners’ ratings of speech production accuracy before and after treatment. However, conventional methods for obtaining these ratings can be time-consuming and frustrating. Crowdsourcing platforms like Amazon Mechanical Turk provide immediate access to a huge pool of potential raters, and our results suggest that by aggregating responses across a large number of nonexpert listeners, we can obtain speech ratings that are comparable in quality to trained listeners’ judgments.
Nightingale, C., Swartz., M. T., Ramig, L. O., & McAllister, T. (2016). Acoustic measures versus crowdsourced listeners’ ratings: Evaluating the efficacy of treatment for hypokinetic dysarthria. Poster presented at the 5th Joint Meeting Acoustical Society of America and Acoustical Society of Japan, Honolulu, HI. Link to manuscript preprint and associated text/code
Harel, D., Hitchcock, E. R., Szeredi, D., Ortiz, J., & McAllister Byun, T. (2016). Finding the experts in the crowd: Accuracy and reliability in crowdsourced measures of children’s covert contrasts. Clinical Linguistics and Phonetics. doi: 10.3109/02699206.2016.1174306