You Only Need One Model for Open-domain Question Answering

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3047 - 3060 December 7-11, 2022 ©2022 Association for Computational Linguistics You Only Need One Model for Open-domain Question Answering Haejun Lee♣ Akhil Kedia♣ Jongwon Lee♦* Ashwin Paranjape♠ Christopher D. Manning♠ Kyoung-Gu Woo♥* ♣ Samsung Research ♦ Samsung Electronics ♠ Stanford University ♥ Growdle Corporation {haejun82.lee, akhil.kedia, jay722.lee}@samsung.com {ashwinp, manning}@cs.stanford.edu, epigramwoo@growdle.com Abstract Recent approaches to Open-domain Question Answering refer to an external knowledge base using a retriever model, optionally rerank pas- sages with a separate reranker model and gen- erate an answer using another reader model. Despite performing related tasks, the mod- els have separate parameters and are weakly- coupled during training. We propose cast- ing the retriever and the reranker as internal passage-wise attention mechanisms applied se- quentially within the transformer architecture and feeding computed representations to the reader, with the hidden representations pro- gressively refined at each stage. This allows us to use a single question answering model trained end-to-end, which is a more efficient use of model capacity and also leads to bet- ter gradient flow. We present a pre-training method to effectively train this architecture and evaluate our model on the Natural Ques- tions and TriviaQA open datasets. For a fixed parameter budget, our model outperforms the previous state-of-the-art model by 1.0 and 0.7 exact match scores. 1 Introduction Open-domain Question Answering (Open QA) is a knowledge-intensive task that finds the answer for the given question from a large-scale knowledge corpus that can easily surpass millions of docu- ments. Thus, how to store and refer to the knowl- edge at such scales is important in terms of both performance and scalability for Open QA systems. Traditional systems rely on information retrieval en- gines such as Lucene. These score the relevance of knowledge to a given query by lexical overlaps be- tween them in a sparse representation space based on TF-IDF or BM25 (Chen et al., 2017; Wang et al., 2018; Yang et al., 2019). However, recent advances in neural language modeling have enabled two new *Work done while at Samsung Research lines of approach; 1) referring to internal knowl- edge parameterized in the model (Brown et al., 2020; Petroni et al., 2019; Roberts et al., 2020), and 2) referring to external knowledge retrieved by matching query and knowledge in dense represen- tation spaces (Karpukhin et al., 2020; Lee et al., 2019; Guu et al., 2020; Lewis et al., 2020; Izacard and Grave, 2021b). Despite the simplicity of the approach, para- metric models have limitations such as a large number of model parameters that require large compute for both training and inference and non- expandable knowledge without re-training. Their implicit knowledge reference also makes it hard to find supporting knowledge and often results in hal- lucinations (Shuster et al., 2021). The current dense retrieval models have advantages over parametric models on these issues (Karpukhin et al., 2020; Guu et al., 2020; Lewis et al., 2020). But most re- trieval models only have a weak …

Content truncated to 500 words