Plot-guided Adversarial Example Construction for Evaluating Open-domain Story Generation

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4334–4344 June 6–11, 2021. ©2021 Association for Computational Linguistics 4334 Plot-guided Adversarial Example Construction for Evaluating Open-domain Story Generation Sarik Ghazarian,1 Zixi Liu,1 Akash SM,2 Ralph Weischedel,1 Aram Galstyan,1 Nanyun Peng1, 3 1University of Southern California / Information Sciences Institute 2Indian Institute of Technology Roorkee 3Computer Science Department of University of California, Los Angeles {sarik, zixiliu, weisched, galstyan}@isi.edu, akashsm@ce.iitr.ac.in, violetpeng@cs.ucla.edu Abstract With the recent advances of open-domain story generation, the lack of reliable automatic eval- uation metrics becomes an increasingly imper- ative issue that hinders the fast development of story generation. According to conducted researches in this regard, learnable evaluation metrics have promised more accurate assess- ments by having higher correlations with hu- man judgments. A critical bottleneck of ob- taining a reliable learnable evaluation metric is the lack of high-quality training data for clas- sifiers to efficiently distinguish plausible and implausible machine-generated stories. Previ- ous works relied on heuristically manipulated plausible examples to mimic possible system drawbacks such as repetition, contradiction, or irrelevant content in the text level, which can be unnatural and oversimplify the characteris- tics of implausible machine-generated stories. We propose to tackle these issues by generat- ing a more comprehensive set of implausible stories using plots, which are structured repre- sentations of controllable factors used to gener- ate stories. Since these plots are compact and structured, it is easier to manipulate them to generate text with targeted undesirable proper- ties, while at the same time maintain the gram- matical correctness and naturalness of the gen- erated sentences. To improve the quality of generated implausible stories, we further apply the adversarial filtering procedure presented by Zellers et al. (2018) to select a more nu- anced set of implausible texts. Experiments show that the evaluation metrics trained on our generated data result in more reliable au- tomatic assessments that correlate remarkably better with human judgments compared to the baselines. 1 Introduction The surge of downstream applications for open- domain natural language generation (NLG), such as dialog systems (Zhang et al., 2020) and story Human Written Story: jenny liked fresh fish. she decided to go fishing to catch her own. she brought her worms and pole and a chair. she sat there all day but didn't catch anything. she packed it up and went home disappointed. Sentence Manipulation: jenny liked fresh fish. she decided to go fishing to catch her own. she wrote songs every single day. she sat there all day but didn't catch anything. she packed it up and went home disappointed. Keyword Manipulation: jenny liked fresh fish. she decided to go fishing to catch her own. she brought her worms and pole and a chair. she sat there all day but didn't catch anything. she unpacked it up and went home disappointed. UNION: jenny liked fresh fish. jim has a very structured workout program to help him achieve goals. she brought her worms and pole and …

Content truncated to 500 words