Choose Your Own Adventure: Paired Suggestions in Collaborative Writing for Evaluating Story Generation Models

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3566–3575 June 6–11, 2021. ©2021 Association for Computational Linguistics 3566 Choose Your Own Adventure: Paired Suggestions in Collaborative Writing for Evaluating Story Generation Models Elizabeth Clark1 and Noah A. Smith1,2 1Paul G. Allen School of Computer Science & Engineering, University of Washington 2Allen Institute for Artificial Intelligence {eaclark7,nasmith}@cs.washington.edu Abstract Story generation is an open-ended and subjec- tive task, which poses a challenge for eval- uating story generation models. We present CHOOSE YOUR OWN ADVENTURE, a collab- orative writing setup for pairwise model eval- uation. Two models generate suggestions to people as they write a short story; we ask writ- ers to choose one of the two suggestions, and we observe which model’s suggestions they prefer. The setup also allows further analysis based on the revisions people make to the sug- gestions. We show that these measures, com- bined with automatic metrics, provide an in- formative picture of the models’ performance, both in cases where the differences in genera- tion methods are small (nucleus vs. top-k sam- pling) and large (GPT2 vs. Fusion models). 1 Introduction Systems that automatically generate text sugges- tions to human authors have emerged as a new application of natural language generation models. Evaluating such models, however, is challenging. Typically, writers rate a single system’s quality af- ter some period of use, for example while author- ing an entire story or poem (e.g., Clark et al., 2018; Ghazvininejad et al., 2017). A model’s quality is measured using Likert scale scores, sometimes combined with additional analysis, like the type or quantity of writer edits (e.g., Roemmele and Gordon, 2015; Akoury et al., 2020). In contrast, a pairwise system evaluation— where evaluators are given two suggestions at the same time and asked to choose between them— would allow researchers to compare generation models directly. Comparative evaluations have been shown to produce more reliable and consistent results than Likert-scale ratings (Callison-Burch et al., 2007; Kiritchenko and Mohammad, 2017), and they have been used to evaluate natural lan- guage generation systems for translation and dia- logue (Otani et al., 2016; Sedoc et al., 2019). Figure 1: CYOA has a writer write a line of the story alone, and then two models generate suggestions for the next line. The writer chooses one (in this case, MODEL1), edits it, and then adds it to the story. They re- peat this process 5 times. CYOA collects writers’ pref- erences between the two models, along with the human- authored, machine-generated, and human-edited text, to evaluate the models. We propose CHOOSE YOUR OWN ADVENTURE (CYOA), a protocol for pairwise evaluations of col- laborative writing models, focusing on story gener- ation. Instead of scoring a single model, we com- pare two models. At fixed points during the writ- ing process, each generates a suggestion, and writ- ers choose one to continue their story (see Fig. 1). The result is utterance-level feedback on which model’s generated text writers …

Content truncated to 500 words