How to Make Neural Natural Language Generation as Reliable as Templates in Task-Oriented Dialogue

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 2877–2888, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 2877 How to Make Neural Natural Language Generation as Reliable as Templates in Task-Oriented Dialogue Henry Elder ADAPT Centre Dublin City University henry.elder@adaptcentre.ie Alexander O’Connor Autodesk, inc. alex.oconnor@autodesk.com Jennifer Foster School of Computing Dublin City University jennifer.foster@dcu.ie Abstract Neural Natural Language Generation (NLG) systems are well known for their unreliabil- ity. To overcome this issue, we propose a data augmentation approach which allows us to re- strict the output of a network and guarantee reliability. While this restriction means gen- eration will be less diverse than if randomly sampled, we include experiments that demon- strate the tendency of existing neural genera- tion approaches to produce dull and repetitive text, and we argue that reliability is more im- portant than diversity for this task. The system trained using this approach scored 100% in se- mantic accuracy on the E2E NLG Challenge dataset, the same as a template system. 1 Introduction The goal of task oriented dialogue is to help a user achieve a narrow goal, such as booking a restaurant or movie ticket. The final step of a conversational interface is generating a response to the user; more specifically, performing surface realization of some structured data containing relevant information. Research into neural NLG systems for the sur- face realization task is popular because such sys- tems may have advantages over the dominant rule and template-based systems: neural NLG systems trained on datasets may be both easier to maintain and to scale to new domains, as well as generat- ing more natural responses (Wen et al., 2015; Guo and Zhao, 2017). But neural NLG systems are not without problems. They are widely considered too unreliable for business applications; they have a tendency to hallucinate facts, unsupported by the structured data they were given (Wiseman et al., 2017). A less well known issue is the template-like gen- eration of neural NLG systems (Wei et al., 2019). Figure 1 highlights this issue; neural NLG systems (TGen and Slug2Slug) are far less diverse than the Figure 1: Surface forms used to express the attribute- value pair: PriceRange[Cheap]. SF not appearing: no surface form was found in the utterance, 73 Remaining SFs: surface forms other than cheap and cheap price range training data (E2E Dataset) in their usage of sur- face forms that express an attribute. Intuitively, one might expect that a neural NLG system trained on a dataset with 75 different surface forms to ex- press an attribute would use a wide variety of them – instead we see only the top two most common surface forms in use. We highlight the issue of lack of diversity, not to provide a specific solution to it, but rather to pro- vide some context for our proposal which relates to reliability of neural NLG systems. Given that our main goal is reliability, we wondered if there were some way to lean into the blandness and …

Content truncated to 500 words