Plug-and-Play Adaptation for Continuously-updated QA

Findings of the Association for Computational Linguistics: ACL 2022, pages 438 - 447 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Plug-and-Play Adaptation for Continuously-updated QA Kyungjae Lee5 Wookje Han1 Seung-won Hwang1∗ Hwaran Lee2 Joonsuk Park2,4 Sang-Woo Lee2,3 1Seoul National University 2NAVER AI Lab 3NAVER CLOVA 4University of Richmond 5LG AI Research Abstract Language models (LMs) have shown great po- tential as implicit knowledge bases (KBs). And for their practical use, knowledge in LMs need to be updated periodically. However, existing tasks to assess LMs’ efficacy as KBs do not ad- equately consider multiple large-scale updates. To this end, we first propose a novel task— Continuously-updated QA (CuQA)—in which multiple large-scale updates are made to LMs, and the performance is measured with respect to the success in adding and updating knowl- edge while retaining existing knowledge. We then present LMs with plug-in modules that ef- fectively handle the updates. Experiments con- ducted on zsRE QA and NQ datasets show that our method outperforms existing approaches. We find that our method is 4x more effective in terms of updates/forgets ratio, compared to a fine-tuning baseline. 1 Introduction LM-as-KB is a new paradigm in which pre-trained language models (LMs) are used as implicit knowl- edge bases (KBs) (Petroni et al., 2019). This is made possible by LMs’ impressive ability to memo- rize factual knowledge (Heinzerling and Inui, 2021; Brown et al., 2020). Recently, two tasks have been used to assess such ability: LAMA, a knowl- edge probing benchmark, challenges LMs to fill in masked words over relational knowledge (Petroni et al., 2019); and closed-book QA (CBQA) ex- amines whether LMs can correctly answer natural language questions (Roberts et al., 2020). For practical usage, LM-as-KB requires that LMs are updated periodically to stay current with the ever-evolving world. Thus, LMs’ ability to update knowledge should also be evaluated. To this end, we present Continuously-updated QA (CuQA), which tests the ability to continuously inject knowledge to update (or target knowledge), ∗correspond to seungwonh@snu.ac.kr T5 Before / After Updates Q3: Who is the most paid player in EPL? A: Alexis Sanchez A: Cristiano Ronaldo Q2: How many episodes re there in Squid Game? A: 9 No answer A: October 28 1955 Q1: When is the birthday of Bill Gates? A: October 28 1955 Source Knowledge Target Knowledge Source Knowledge Target Knowledge … - Scenario 1: inject new knowledge - Scenario 2 : update existing knowledge A: October 28 1955 Q1: When is the birthday of Bill Gates? Q3: How many episodes are there in Squid Game? A: 9 Q2: Who is the most paid player in EPL? A: Cristiano Ronaldo Q2: Who is the most paid player in EPL? A: Alexis Sanchez Figure 1: Examples of CuQA showcasing two scenarios. while retaining existing knowledge (or source knowledge). Specifically, we consider multiple large-scale knowledge updates (8k to 60k) covering two scenarios: injecting new knowledge (Scenario 1 in Figure 1) and updating existing knowledge (Scenario 2 in Figure 1) . Our goal is to organize the implicit …

Content truncated to 500 words