Designing a Minimal Retrieve-and-Read System for Open-Domain Question Answering

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5856–5865 June 6–11, 2021. ©2021 Association for Computational Linguistics 5856 Designing a Minimal Retrieve-and-Read System for Open-Domain Question Answering Sohee Yang1˚and Minjoon Seo1,2 1KAIST AI 2NAVER Corp. {sohee.yang,minjoon}@kaist.ac.kr Abstract In open-domain question answering (QA), retrieve-and-read mechanism has the inherent benefit of interpretability and the easiness of adding, removing, or editing knowledge com- pared to the parametric approaches of closed- book QA models. However, it is also known to suffer from its large storage footprint due to its document corpus and index. Here, we discuss several orthogonal strategies to dras- tically reduce the footprint of a retrieve-and- read open-domain QA system by up to 160x. Our results indicate that retrieve-and-read can be a viable option even in a highly constrained serving environment such as edge devices, as we show that it can achieve better accuracy than a purely parametric model with compara- ble docker-level system size.1 1 Introduction Open-domain question answering (QA) is the task of finding answers to generic factoid questions. In recent literature, the task is largely approached in two ways, namely retrieve & read and paramet- ric. The former solves the problem by first re- trieving documents relevant to the question from a large knowledge source and then reading the re- trieved documents to find out the answer (Lee et al., 2019; Guu et al., 2019; Karpukhin et al., 2020; Lewis et al., 2020; Izacard and Grave, 2021). The latter, also known as closed-book QA, generates the answer in a purely parametric end-to-end man- ner (Brown et al., 2020; Roberts et al., 2020). While a parametric model enjoys the benefit in terms of system size that they do not require ad- ditional knowledge source like a retrieve & read system does, its fundamental limitations are that their predictions are not so interpretable and they are not suitable for dynamic knowledge source as ˚ Most of the work was done while the author was work- ing at NAVER Corp. 1Our code and model weights are available in https://github.com/clovaai/minimal-rnr-qa. End-to-End QA Accuracy (EM) DPR §2.1 Passage Filtering §2.2.1 128D Embedding §2.2.1 MobileBERT §2.2.2 Retriever Encoder Sharing §2.2.3 Unified Retriever- Reader Through KD §2.2.4 Iterative Finetuning §2.3 Post-Training Compression (Minimal R&R) T5-1.1-small+SSM T5-1.1-XL+SSM T5-1.1-XXL+SSM 15 20 25 30 35 40 0 1 2 3 4 5 6 7 8 End-to-End QA Accuracy (EM) System Footprint (GB) Minimal R&R T5 w/ Compression 78 Figure 1: System footprint vs. Exact Match (EM) accuracy on EfficientQA dev set. System footprint is measured by the command du -h / inside the standalone docker container as stated in the EfficientQA competition guideline. The red plot from left to right shows a path of reducing the size of an open- domain QA system with DPR from 77.5GB to 484.68MB by successively applying each of the strategies in Section 2. The storage footprints of the baseline systems with T5 are calculated assuming the use of the lightweight docker …

Content truncated to 500 words