Detecting Hallucinated Content in Conditional Neural Sequence Generation

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 1393–1404 August 1–6, 2021. ©2021 Association for Computational Linguistics 1393 Detecting Hallucinated Content in Conditional Neural Sequence Generation Chunting Zhou1∗, Graham Neubig1, Jiatao Gu2, Mona Diab2, Francisco Guzm´an2, Luke Zettlemoyer2, Marjan Ghazvininejad2 Language Technologies Institute, Carnegie Mellon University1 Facebook AI Research2 {chuntinz,gneubig}@cs.cmu.edu, {jgu,mdiab,fguzman,lsz,ghazvini}@fb.com Abstract Neural sequence models can generate highly fluent sentences, but recent studies have also shown that they are also prone to hallucinate additional content not supported by the input. These variety of fluent but wrong outputs are particularly problematic, as it will not be possi- ble for users to tell they are being presented in- correct content. To detect these errors, we pro- pose a task to predict whether each token in the output sequence is hallucinated (not contained in the input) and collect new manually anno- tated evaluation sets for this task. We also in- troduce a method for learning to detect halluci- nations using pretrained language models fine tuned on synthetic data that includes automat- ically inserted hallucinations. Experiments on machine translation (MT) and abstractive sum- marization demonstrate that our proposed ap- proach consistently outperforms strong base- lines on all benchmark datasets. We further demonstrate how to use the token-level halluci- nation labels to define a fine-grained loss over the target sequence in low-resource MT and achieve significant improvements over strong baseline methods. We also apply our method to word-level quality estimation for MT and show its effectiveness in both supervised and unsupervised settings 1. 1 Introduction Neural sequence models for tasks such as data-to- text generation (Puduppully et al., 2019), machine translation (MT; Vaswani et al. (2017); Wu et al. (2016)) and text summarization (Rothe et al., 2020) can often generate fluent text that is sometimes preferred to human-written content (L¨aubli et al., 2018; Brown et al., 2020). However, they also often generate texts that lack global logical consis- ∗Most work was done during an internship at FAIR. 1Codes and data available at https://github.com/ violet-zct/fairseq-detect-hallucination. Jerry with 1 1 1 happily 1 (Source meaning: Mike goes to the bookstore on Thursday. ) 1 goes to the bookstore Machine Translation 0 0 0 0 迈克周四 去书店. 。 Source Input his friend. Figure 1: An example of token-level hallucination de- tection from MT. The grey box is an example of MT output and the labels above indicate if each word is faithful (0) to the input or hallucinated (1). tency (Marcus and Davis, 2020), are dull and repet- itive (Welleck et al., 2019), or contain hallucinated content that is not entailed by the input (Maynez et al., 2020; Martindale et al., 2019). In this paper, we focus on tackling the latter problem, aiming to automatically identify and quantify content in the output that is not faithful to the input text. The risk of generating unfaithful content im- pedes the safe deployment of neural sequence gen- eration models. The first step to building models that do not suffer from these failures is the assess- ment and identification …

Content truncated to 500 words