StoryER: Automatic Story Evaluation via Ranking, Rating and Reasoning

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1739 - 1753 December 7-11, 2022 ©2022 Association for Computational Linguistics StoryER: Automatic Story Evaluation via Ranking, Rating and Reasoning Hong Chen1,3, Duc Minh Vo1, Hiroya Takamura2,3, Yusuke Miyao1,3, Hideki Nakayama1,3 The University of Tokyo1, Tokyo Institute of Technology2 National Institute of Advanced Industrial Science and Technology, Japan3 {chen, vmduc, nakayama}@nlab.ci.i.u-tokyo.ac.jp yusuke@is.s.u-tokyo.ac.jp takamura.hiroya@aist.go.jp Abstract Existing automatic story evaluation methods place a premium on story lexical level coher- ence, deviating from human preference. We go beyond this limitation by considering a novel Story Evaluation method that mimics hu- man preference when judging a story, namely StoryER, which consists of three sub-tasks: Ranking, Rating and Reasoning. Given either a machine-generated or a human-written story, StoryER requires the machine to output 1) a preference score that corresponds to human preference, 2) specific ratings and their corre- sponding confidences and 3) comments for var- ious aspects (e.g., opening, character-shaping). To support these tasks, we introduce a well- annotated dataset comprising (i) 100k ranked story pairs; and (ii) a set of 46k ratings and com- ments on various aspects of the story. We fine- tune Longformer-Encoder-Decoder (LED) on the collected dataset, with the encoder respon- sible for preference score and aspect prediction and the decoder for comment generation. Our comprehensive experiments result in a com- petitive benchmark for each task, showing the high correlation to human preference. In addi- tion, we have witnessed the joint learning of the preference scores, the aspect ratings, and the comments brings gain in each single task. Our dataset and benchmarks are publicly avail- able to advance the research of story evaluation tasks.1 1 Introduction Even for humans, evaluating story quality is a chal- lenging task. Although many literature criteria have been proposed, the most straightforward way is to count how many readers like the story which is re- ferred as to human preference. Bearing it in mind, story writing community usually uses upvote count 1Dataset and pre-trained model demo are available at anony- mous website http://storytelling-lab.com/ eval and https://github.com/sairin1202/ StoryER Figure 1: The existing story evaluation method (UNION) outputs a score for estimating the coherence of the stories, while human-written stories rarely suffer from this problem. Our model (Ours) which is trained by comparing two stories (Ranking), evaluates the story based on human preference (i.e., upvote counts), pro- duces scores for various aspects (Rating), and leaves comments (Reasoning). Our model is applicable to both machine-generated and human-written stories. as a story quality criterion. As shown in Fig. 1, more readers like the left story (upvote count = 1.8k) rather than the right one (upvote count = 1). Existing methods which use referenced metrics (e.g., BLEU (Papineni et al., 2002), METEOR (Banerjee and Lavie, 2005), ROUGE (Lin, 2004)) and unreferenced met- rics (e.g., UNION (Guan and Huang, 2020), MANPLTS (Ghazarian et al., 2021)), deviate from human preference (Fig. 1). On the contrary, we aim to explicitly evaluate a story, introducing a human preference-liked system consisting of three …

Content truncated to 500 words