OpenMEVA: A Benchmark for Evaluating Open-ended Story Generation Metrics

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 6394–6407 August 1–6, 2021. ©2021 Association for Computational Linguistics 6394 OpenMEVA: A Benchmark for Evaluating Open-ended Story Generation Metrics Jian Guan1, Zhexin Zhang1, Zhuoer Feng1, Zitao Liu2, Wenbiao Ding2, Xiaoxi Mao3, Changjie Fan3 and Minlie Huang1∗ 1The CoAI group, DCST; 1Institute for Artificial Intelligence; 1State Key Lab of Intelligent Technology and Systems; 1Beijing National Research Center for Information Science and Technology; 1Tsinghua University, Beijing 100084, China. 2TAL Education Group. 3Netease Fuxi AI Lab. {j-guan19,zx-zhang18,fze17}@mails.tsinghua.edu.cn, zitao.jerry.liu@gmail.com, dingwenbiao@100tal.com, {maoxiaoxi,fanchangjie}@corp.netease.com, aihuang@tsinghua.edu.cn Abstract Automatic metrics are essential for developing natural language generation (NLG) models, particularly for open-ended language genera- tion tasks such as story generation. However, existing automatic metrics are observed to cor- relate poorly with human evaluation. The lack of standardized benchmark datasets makes it difficult to fully evaluate the capabilities of a metric and fairly compare different metrics. Therefore, we propose OpenMEVA, a bench- mark for evaluating open-ended story genera- tion metrics. OpenMEVA provides a compre- hensive test suite to assess the capabilities of metrics, including (a) the correlation with hu- man judgments, (b) the generalization to dif- ferent model outputs and datasets, (c) the abil- ity to judge story coherence, and (d) the ro- bustness to perturbations. To this end, Open- MEVA includes both manually annotated sto- ries and auto-constructed test examples. We evaluate existing metrics on OpenMEVA and observe that they have poor correlation with human judgments, fail to recognize discourse- level incoherence, and lack inferential knowl- edge (e.g., causal order between events), the generalization ability and robustness. Our study presents insights for developing NLG models and metrics in further research. 1 Introduction Significant advances have been witnessed in many NLG tasks with pretraining models (Devlin et al., 2019; Brown et al., 2020). However, existing gen- eration models are still far behind the human-level performance to generate reasonable texts, particu- larly for open-ended generation tasks such as story generation (Fan et al., 2018; Guan et al., 2020). One critical obstacle is the lack of powerful met- rics for measuring the quality of generation. ∗Corresponding author The standard paradigm for evaluating NLG met- rics is to calculate the correlation with human judg- ments on manually annotated datasets (Tao et al., 2018; Sellam et al., 2020). Recent studies have discovered that the existing automatic metrics may correlate poorly with human judgments (Liu et al., 2016; Guan and Huang, 2020). Unfortunately, the lack of benchmark datasets makes it challenging to completely assess the capabilities of a metric and fairly compare different metrics. Firstly, anno- tated datasets usually contain innate data bias and annotation bias. Secondly, summarizing the per- formance with a single aggregate statistic (e.g., a correlation score) makes it difficult to probe which aspects a metric can successfully capture and which can not. Therefore, many alternative approaches have been proposed to evaluate NLG metrics, such as measuring the robustness to adversarial exam- ples (Zhang* et al., 2020), and the …

Content truncated to 500 words