A Simple Recipe towards Reducing Hallucination in Neural Surface Realisation

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2673–2679 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 2673 A Simple Recipe towards Reducing Hallucination in Neural Surface Realisation Feng Nie1∗ Jin-Ge Yao2 Jinpeng Wang2 Rong Pan1 Chin-Yew Lin2 1Sun Yat-Sen University 2Microsoft Research Asia 1fengniesysu@gmail.com, 1panr@sysu.edu.cn 2{jinge.yao, jinpwa, cyl}@microsoft.com Abstract Recent neural language generation systems of- ten hallucinate contents (i.e., producing irrel- evant or contradicted facts), especially when trained on loosely corresponding pairs of the input structure and text. To mitigate this is- sue, we propose to integrate a language under- standing module for data refinement with self- training iterations to effectively induce strong equivalence between the input data and the paired text. Experiments on the E2E chal- lenge dataset show that our proposed frame- work can reduce more than 50% relative un- aligned noise from the original data-text pairs. A vanilla sequence-to-sequence neural NLG model trained on the refined data has improved on content correctness compared with the cur- rent state-of-the-art ensemble generator. 1 Introduction Neural models for natural language generation (NLG) based on the encoder-decoder framework have become quite popular recently (Wen et al., 2015; Mei et al., 2016; Wiseman et al., 2017; Wen et al., 2017; Chisholm et al., 2017; Nie et al., 2018, inter alia). Albeit being appealing for producing fluent and diverse sentences, neural NLG models often suffer from a severe issue of content halluci- nation (Reiter, 2018a), which refers to the problem that the generated texts often contain information that is irrelevant to or contradicted with the input. Given that similar issues have been less reported or noticed in the latest neural machine translation systems, we believe that the origin of the issue for neural NLG comes from the data side. Current datasets used for training neural NLG systems of- ten include instances that do not contain the same amount of information from the input structure and the output text (Perez-Beltrachini and Gar- dent, 2017). There is no exception for datasets ∗Contribution during internship at Microsoft. MR Name Rating Price Golden Palace 5 out of 5 Cheap Reference: Golden Palace is a restaurant specializing in breakfast in the low price range. Table 1: A loosely corresponded MR-text pair. Bolded phrases conforms to the MR, underlined words are domain-specific additional information, and italic val- ues in the MR are not realised in the reference. originally intended for surface realisation (“how to say”) without focusing on content selection (“what to say”). Table 1 depicts an example, where the attribute Rating=5 out of 5 in the in- put meaning representation (MR) is not verbalised in a reference text written by human, while the word restaurant in the reference should refer to an attribute value EatType=Restaurant not contained in the MR. Without explicit alignments in between MRs and the corresponding utterances for guidance, neural systems trained on such data often produce unexpected errors. Previous work attempted at injecting indirect semantic control over the encoder-decoder archi- tecture (Wen et …

Content truncated to 500 words