DisentQA: Disentangling Parametric and Contextual Knowledge with Counterfactual Question Answering

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 10056–10070 July 9-14, 2023 ©2023 Association for Computational Linguistics DisentQA: Disentangling Parametric and Contextual Knowledge with Counterfactual Question Answering Ella Neeman1 Roee Aharoni2 Or Honovich3 Leshem Choshen1 Idan Szpektor2 Omri Abend1 1The Hebrew University of Jerusalem 2Google Research 3Tel Aviv University {ella.neeman, leshem.choshen, omri.abend}@mail.huji.ac.il or.honovich@gmail.com {roeeaharoni,szpektor}@google.com Abstract Question answering models commonly have access to two sources of “knowledge” during inference time: (1) parametric knowledge – the factual knowledge encoded in the model weights, and (2) contextual knowledge – ex- ternal knowledge (e.g., a Wikipedia passage) given to the model to generate a grounded an- swer. Having these two sources of knowledge entangled together is a core issue for genera- tive QA models as it is unclear whether the answer stems from the given non-parametric knowledge or not. This unclarity has impli- cations on issues of trust, interpretability and factuality. In this work, we propose a new paradigm in which QA models are trained to disentangle the two sources of knowledge. Us- ing counterfactual data augmentation, we in- troduce a model that predicts two answers for a given question: one based on given contex- tual knowledge and one based on parametric knowledge. Our experiments on the Natural Questions dataset show that this approach im- proves the performance of QA models by mak- ing them more robust to knowledge conflicts between the two knowledge sources, while generating useful disentangled answers. 1 Introduction Question answering (QA) systems are important in many real-world scenarios that require quick access to large bodies of knowledge like the web. Much of the recent progress on QA stems from using pretrained models, shown to implicitly store knowl- edge in their parameters (Roberts et al., 2020). As a result, QA models have access to two knowledge sources when generating an answer: (1) parametric knowledge – knowledge encoded (or “memorized”) in the model parameters, and (2) contextual knowledge – knowledge encapsulated within external textual sources given to the model at inference time as the context of the question, such as paragraphs retrieved based on the question. Question: Who is the guy on Keeping Up with the Kardashians? Factual Context: Jonathan Cheban (born c. 1974) is a reality - television star and entrepreneur. He is noted for his recurring role on the show Keeping Up with the Kardashians and its spinoffs. Contextual Answer: Jonathan Cheban Parametric Answer: Scott Disick Counterfactual Context: Jason Momoa (born c. 1974) is a reality - television star and entrepreneur. He is noted for his recurring role on the show Keeping Up with the Kardashians and its spinoffs. Contextual Answer: Jason Momoa Parametric Answer: Kanye West Figure 1: Example outputs from our disentangled QA model on the Natural Questions dataset. The model generates two answers at once – one based on the given context (blue and red), and another based on its para- metric knowledge (green). Jonathan Cheban, Scott Dis- ick and Kanye West are all prominent male characters on …

Content truncated to 500 words