Understanding and Detecting Hallucinations in Neural Machine Translation via Model Introspection

Understanding and Detecting Hallucinations in Neural Machine Translation via Model Introspection Weijia Xu Microsoft Research, Redmond, USA weijiaxu@microsoft.com Sweta Agrawal University of Maryland, USA sweagraw@cs.umd.edu Eleftheria Briakou University of Maryland, USA ebriakou@cs.umd.edu Marianna J. Martindale University of Maryland, USA mmartind@umd.edu Marine Carpuat University of Maryland, USA marine@cs.umd.edu Abstract Neural sequence generation models are known to ‘‘hallucinate’’, by producing outputs that are unrelated to the source text. These hallucina- tions are potentially harmful, yet it remains unclear in what conditions they arise and how to mitigate their impact. In this work, we first identify internal model symptoms of hal- lucinations by analyzing the relative token contributions to the generation in contrastive hallucinated vs. non-hallucinated outputs gen- erated via source perturbations. We then show that these symptoms are reliable indicators of natural hallucinations, by using them to de- sign a lightweight hallucination detector which outperforms both model-free baselines and strong classifiers based on quality estimation or large pre-trained models on manually an- notated English-Chinese and German-English translation test beds. 1 Introduction While neural language generation models can gen- erate high quality text in many settings, they also fail in counter-intuitive ways, for instance by ‘‘hallucinating’’ (Wiseman et al., 2017; Lee et al., 2018; Falke et al., 2019). In the most se- vere case, known as ‘‘detached hallucinations’’ (Raunak et al., 2021), the output is completely detached from the source, which not only reveals fundamental limitations of current models, but also risks misleading users and undermining trust (Bender et al., 2021; Martindale and Carpuat, 2018). Yet, we lack a systematic understanding of the conditions where hallucinations arise, as hallucinations occur infrequently among transla- tions of naturally occurring text. As a workaround, prior work has largely focused on black-box de- tection methods which train neural classifiers on synthetic data constructed by heuristics (Falke et al., 2019; Zhou et al., 2021), and on studying hallucinations given artificially perturbed inputs (Lee et al., 2018; Shi et al., 2022). In this paper, we address the problem by first identifying the internal model symptoms that characterize hallucinations given artificial inputs and then testing the discovered symptoms on translations of natural texts. Specifically, we study hallucinations in Neural Machine Trans- lation (NMT) using two types of interpretability techniques: saliency analysis and perturbations. We use saliency analysis (Bach et al., 2015; Voita et al., 2021) to compare the relative contributions of various tokens to the hallucinated vs. non- hallucinated outputs generated by diverse adver- sarial perturbations in the inputs (Table 1) inspired by Lee et al. (2018) and Raunak et al. (2021). Re- sults surprisingly show that source contribution patterns are stronger indicators of hallucinations than the relative contributions of the source and target, as had been previously hypothesized (Voita et al., 2021). We discover two distinctive source contribution patterns, including 1) con- centrated contribution from a small subset of source tokens, and 2) the staticity of the source contribution distribution along the generation steps (§ 3). We further show that the symptoms identified generalize to hallucinations on natural inputs …

Content truncated to 500 words