On the Origin of Hallucinations in Conversational Models: Is it the Datasets or the Models?

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5271 - 5285 July 10-15, 2022 ©2022 Association for Computational Linguistics On the Origin of Hallucinations in Conversational Models: Is it the Datasets or the Models? Nouha Dziri† ♢ Sivan Milton‡ Mo Yu¶ Osmar Zaiane† Siva Reddy♢ ‡ †University of Alberta ♢Mila – Quebec AI Institute ‡McGill University ¶IBM Research dziri@cs.ualberta.ca Abstract Knowledge-grounded conversational models are known to suffer from producing factually invalid statements, a phenomenon commonly called hallucination. In this work, we investi- gate the underlying causes of this phenomenon: is hallucination due to the training data, or to the models? We conduct a comprehen- sive human study on both existing knowledge- grounded conversational benchmarks and sev- eral state-of-the-art models. Our study reveals that the standard benchmarks consist of >60% hallucinated responses, leading to models that not only hallucinate but even amplify halluci- nations. Our findings raise important questions on the quality of existing datasets and models trained using them. We make our annotations publicly available for future research.1 1 Introduction Knowledge-grounded conversational models, pow- ered by large pre-trained language models (Radford et al., 2019; Brown et al., 2020; Raffel et al., 2020), are well-known to generate factually incorrect state- ments, a phenomenon commonly called hallucina- tion (Dziri et al., 2021b; Rashkin et al., 2021b). A large commonality in the majority of prior work seeks to address hallucination by ameliorating the model (Shuster et al., 2021; Mielke et al., 2020; Dziri et al., 2021a; Rashkin et al., 2021b), but no attempt has been made so far to audit the conversa- tional benchmarks to the best of our knowledge. On one hand, knowledge-grounded conversa- tional benchmarks may contain hallucinations due to error-prone collection protocols, or due to a de- sign framework that encourages informativeness over faithfulness. Existing dialogue systems are typically trained on corpora crowd-sourced through online platforms (Dinan et al., 2018; Gopalakrish- nan et al., 2019; Moon et al., 2019). With loose 1https://github.com/McGill-NLP/FaithDial Figure 1: An example of a hallucinated conversation from the Wizard of Wikipedia dataset (Dinan et al., 2018). The wizard (yellow) is hallucinating information that cannot be inferred from the knowledge-snippet: hallucinated subjective content (red) and hallucinated objective content (blue). incentive to come up with faithfully-grounded ut- terances on the provided knowledge, crowdwork- ers may ignore knowledge-snippets altogether, use their personal knowledge or sometimes assume a fictional persona, resulting in conversations that are rife with subjective content and unverified factual knowledge. Figure 1 shows a hallucinated conver- sation from the WOW dataset (Dinan et al., 2018), On the other hand, neural conversational models are not necessarily designed to generate faithful outputs, but to mimic the distributional properties of the data. This kind of optimization will likely push the models to replicate and even amplify the hallucination behaviour at test time (Bender et al., 2021). The presence of even few hallucinated re- sponses may skew the data distribution in a way that curbs the model’s ability to …

Content truncated to 500 words