Of Human Criteria and Automatic Metrics: A Benchmark of the Evaluation of Story Generation

Proceedings of the 29th International Conference on Computational Linguistics, pages 5794–5836 October 12–17, 2022. 5794 Of Human Criteria and Automatic Metrics: A Benchmark of the Evaluation of Story Generation Cyril Chhun1 Pierre Colombo2∗ Fabian M. Suchanek1 Chloé Clavel1 1LTCI, Télécom Paris, Institut Polytechnique de Paris 2Lab of Mathematics and Informatics (MICS), CentraleSupélec, Université Paris-Saclay cyril.chhun@telecom-paris.fr Abstract Research on Automatic Story Generation (ASG) relies heavily on human and automatic evaluation. However, there is no consensus on which human evaluation criteria to use, and no analysis of how well automatic criteria corre- late with them. In this paper, we propose to re-evaluate ASG evaluation. We introduce a set of 6 orthogonal and comprehensive human criteria, carefully motivated by the social sci- ences literature. We also present HANNA, an annotated dataset of 1,056 stories produced by 10 different ASG systems. HANNA allows us to quantitatively evaluate the correlations of 72 au- tomatic metrics with human criteria. Our analy- sis highlights the weaknesses of current metrics for ASG and allows us to formulate practical recommendations for ASG evaluation. 1 Introduction Storytelling is at the heart of human societies: skill- ful storytelling allows a narrator to connect more authentically with their audience and listeners, and to understand the essence of complex concepts better (Suzuki et al., 2018). Numerous applica- tions could benefit from strong automatic story generation systems, including gaming (Hartsook et al., 2011), communication (Alhussain and Azmi, 2021), and education (Aylett et al., 2007). Sev- eral approaches have been explored to generate stories automatically or with minimum editing ef- forts (Alabdulkarim et al., 2021). Automatic story generation (ASG) takes as input a short sentence (a prompt) and aims at generating a narrative from it (Cavazza and Pizzi, 2006; Lebowitz, 1985). Ad- vances in neural language models (Radford et al., 2018, 2019; Brown et al., 2020) have allowed sub- stantial progress in ASG. To further improve the quality of generated stories, it is indispensable to systematically evaluate ASG ∗Previously from Laboratoire des Signaux et Systèmes (L2S), CentraleSupélec, CNRS, Université Paris-Saclay. models. However, there is little work that specifi- cally studies ASG evaluation. Most research works rely on human criteria such as coherence (Xu et al., 2018; Colombo et al., 2019; Jalalzai et al., 2020), relevance (Jhamtani and Berg-Kirkpatrick, 2020), overall quality (Brahman and Chaturvedi, 2020), narrative flow (Rashkin et al., 2020), and creativity (Pascual et al., 2021). However, taken individually, these criteria fail to encompass all aspects of the task, and there is no consensus on a set of crite- ria that would cover those aspects in a complete and non-redundant fashion. Due to the high cost of human annotation, system quality is also often evaluated using automatic metrics. However, it is not clear how these metrics correlate with human judgment in ASG, and thus how suitable they are at all for the evaluation of ASG. Contributions. In this work, we revisit both hu- man and automatic evaluation of ASG. We believe that this meta-evaluation is a missing piece in the ASG literature and a …

Content truncated to 500 words