UNION: An Unreferenced Metric for Evaluating Open-ended Story Generation

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 9157–9166, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 9157 UNION: An Unreferenced Metric for Evaluating Open-ended Story Generation Jian Guan, Minlie Huang∗ Department of Computer Science and Technology, Institute for Artificial Intelligence, State Key Lab of Intelligent Technology and Systems, Beijing National Research Center for Information Science and Technology, Tsinghua University, Beijing 100084, China j-guan19@mails.tsinghua.edu.cn, aihuang@tsinghua.edu.cn Abstract Despite the success of existing referenced met- rics (e.g., BLEU and MoverScore), they cor- relate poorly with human judgments for open- ended text generation including story or dia- log generation because of the notorious one- to-many issue: there are many plausible out- puts for the same input, which may differ sub- stantially in literal or semantics from the lim- ited number of given references. To allevi- ate this issue, we propose UNION, a learnable UNreferenced metrIc for evaluating Open- eNded story generation, which measures the quality of a generated story without any refer- ence. Built on top of BERT, UNION is trained to distinguish human-written stories from neg- ative samples and recover the perturbation in negative stories. We propose an approach of constructing negative samples by mimick- ing the errors commonly observed in existing NLG models, including repeated plots, con- flicting logic, and long-range incoherence. Ex- periments on two story datasets demonstrate that UNION is a reliable measure for evalu- ating the quality of generated stories, which correlates better with human judgments and is more generalizable than existing state-of-the- art metrics. 1 Introduction Significant advances have been witnessed with neu- ral encoder-decoder paradigm (Sutskever et al., 2014), transformer-based architecture (Vaswani et al., 2017) and large-scale pretraining models (De- vlin et al., 2019; Radford et al., 2019) in a wide array of natural language generation (NLG) tasks including machine translation (Bahdanau et al., 2015), story generation (Fan et al., 2018; Guan et al., 2020), and many more. However, the re- search is increasingly hindered by the lack of effec- ∗Corresponding author Leading Context Jack was at the bar. Reference By Human He noticed a phone on the floor. He was going to take it to lost and found. But it started ringing on the way. Jack answered it and returned it to the owner’s friends. Sample 1 (Reasonable, B=0.29, M=0.49, U=1.00) On the way out he noticed a phone on the floor. He asked around if anybody owned it. Eventually he gave it to the bartender. They put it into their lost and found box. Sample 2 (Reasonable, B=0.14, M=0.27, U=1.00) He had a drinking problem. He kept having more beers. After a while he passed out. When he waked up, he was surprised to find that he lost over a hundred dollars. Sample 3 (Unreasonable, B=0.20, M=0.35, U=0.00) He was going to get drunk and get drunk. The bartender told him it was already time to leave. Jack started drinking. Jack wound up returning but cops came on the way home. Table 1: Generated story samples given the same …

Content truncated to 500 words