The Perils of Using Mechanical Turk to Evaluate Open-Ended Text Generation

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1265–1285 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 1265 The Perils of Using Mechanical Turk to Evaluate Open-Ended Text Generation Marzena Karpinska Nader Akoury Mohit Iyyer University of Massachusetts Amherst {mkarpinska,nsa,miyyer}@cs.umass.edu Abstract Recent text generation research has increas- ingly focused on open-ended domains such as story and poetry generation. Because mod- els built for such tasks are difficult to evaluate automatically, most researchers in the space justify their modeling choices by collecting crowdsourced human judgments of text qual- ity (e.g., Likert scores of coherence or gram- maticality) from Amazon Mechanical Turk (AMT). In this paper, we first conduct a sur- vey of 45 open-ended text generation papers and find that the vast majority of them fail to report crucial details about their AMT tasks, hindering reproducibility. We then run a se- ries of story evaluation experiments with both AMT workers and English teachers and dis- cover that even with strict qualification fil- ters, AMT workers (unlike teachers) fail to distinguish between model-generated text and human-generated references. We show that AMT worker judgments improve when they are shown model-generated output alongside human-generated references, which enables the workers to better calibrate their ratings. Finally, interviews with the English teachers provide deeper insights into the challenges of the evaluation process, particularly when rat- ing model-generated text. 1 Introduction Recent advances in neural language modeling have spurred research into open-ended text generation tasks such as story generation (Peng et al., 2018a), style transfer (Krishna et al., 2020), and pun gen- eration (He et al., 2019). Since the space of pos- sible outputs for these tasks is huge compared to more constrained problems such as machine trans- lation, automatic metrics such as BLEU (Papineni et al., 2002) and ROUGE (Lin, 2004) that measure similarity to reference texts are mostly uninforma- tive (Akoury et al., 2020).1 Human evaluation of 1Nevertheless, such metrics are commonly reported in research papers on open-ended text generation. model-generated text, which is critical for open- ended tasks given the unreliability of automatic metrics (Peng et al., 2017; Reiter, 2018; See et al., 2019), is frequently conducted on Amazon’s popu- lar Mechanical Turk platform (AMT) to minimize cost and time. Most existing AMT studies ask crowdworkers to provide Likert scale ratings of various properties of generated text, such as flu- ency and likability. In this paper, we study the reliability and repro- ducibility of AMT evaluations of open-ended text generation. We first conduct a survey of papers on open-ended text generation between 2018-2020 and find many critical details often go unreported (e.g., worker qualifications, payment, task descrip- tions, annotator agreement), a finding in line with prior reproducibility studies outside open-ended text generation (Card et al., 2020; Howcroft et al., 2020; van der Lee et al., 2021). Next, we perform a series of story generation evaluations with both AMT workers and expert raters (English teachers), applying a variant of the most common task configuration that appeared in our survey (5 …

Content truncated to 500 words