Entity-Based Knowledge Conflicts in Question Answering

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7052–7063 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 7052 Entity-Based Knowledge Conflicts in Question Answering Shayne Longpre∗♠ Kartik Perisetla∗♠ Anthony Chen∗♥ Nikhil Ramesh♠ Chris DuBois♠ Sameer Singh♥ ♠Apple ♥University of California, Irvine slongpre@mit.edu {kperisetla, nikhilr, cdubois} @apple.com {anthony.chen, sameer} @uci.edu Abstract Knowledge-dependent tasks typically use two sources of knowledge: parametric, learned at training time, and contextual, given as a pas- sage at inference time. To understand how models use these sources together, we for- malize the problem of knowledge conflicts, where the contextual information contradicts the learned information. Analyzing the be- haviour of popular models, we measure their over-reliance on memorized information (the cause of hallucinations), and uncover impor- tant factors that exacerbate this behaviour. Lastly, we propose a simple method to mit- igate over-reliance on parametric knowledge which minimizes hallucination and improves out-of-distribution generalization by 4%−7%. Our findings demonstrate the importance for practitioners to evaluate model tendency to hal- lucinate rather than read, and show that our mitigation strategy encourages generalization to evolving information (i.e., time-dependent queries). To encourage these practices, we have released our framework for generating knowledge conflicts.1 1 Introduction Knowledge-dependent tasks, such as open-retrieval question answering (QA), require expansive “world knowledge”, common sense, and reasoning abili- ties. State-of-the-art approaches typically follow a retrieve-and-read setup (Chen et al., 2017), where the retriever sources relevant documents, and the reader produces an answer from these. In this sense, there are two sources of knowledge contributing to model inference with an ambiguous and opaque division of labour. The first is the implicit para- metric knowledge (i.e., their learned weights) in- stilled by pre-training and fine-tuning (Petroni et al., 2019). The second is contextual knowledge, usu- ∗Equal Contribution. 1Framework is provided at https://github.com/ apple/ml-knowledge-conflicts. Question: Who did US fight in world war 1? Original Context: The United States declared war on Germany on April 6, 1917, over 2 years after World War I started . . . Original Answer: Germany Model Prediction: Germany Question: Who did US fight in world war 1? Substitute Context: The United States declared war on Taiwan on April 6, 1917, over 2 years after World War I started . . . Substitute Answer: Taiwan Model Prediction: Germany Figure 1: Knowledge Substitution: A substitute ex- ample is derived from the original example by replac- ing the original answer, Germany, with a similar type of answer, i.e. Taiwan. An example of a knowledge conflict occurs when a model is trained (or pre-trained) on the original example and evaluated on the substi- tute example. ally sourced as passages of text from the retriever (Fisch et al., 2019). As a testament to their memorization abilities, large language models can produce competitive results relying only on their own parametric knowl- edge, without access to relevant documents (Brown et al., 2020; Roberts et al., 2020). However, this memorization behaviour has manifested in a pen- chant to hallucinate, or parrot answers memorized during training, completely ignoring relevant docu- ments …

Content truncated to 500 words