Don’t Read Too Much Into It: Adaptive Computation for Open-Domain Question Answering

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 3029–3039, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 3029 Don’t Read Too Much Into It: Adaptive Computation for Open-Domain Question Answering Yuxiang Wu Sebastian Riedel Pasquale Minervini Pontus Stenetorp University College London {yuxiang.wu,s.riedel,p.minervini,p.stenetorp}@cs.ucl.ac.uk Abstract Most approaches to Open-Domain Question Answering consist of a light-weight retriever that selects a set of candidate passages, and a computationally expensive reader that exam- ines the passages to identify the correct answer. Previous works have shown that as the num- ber of retrieved passages increases, so does the performance of the reader. However, they as- sume all retrieved passages are of equal impor- tance and allocate the same amount of com- putation to them, leading to a substantial in- crease in computational cost. To reduce this cost, we propose the use of adaptive compu- tation to control the computational budget al- located for the passages to be read. We first introduce a technique operating on individual passages in isolation which relies on anytime prediction and a per-layer estimation of an early exit probability. We then introduce SKY- LINEBUILDER, an approach for dynamically deciding on which passage to allocate compu- tation at each step, based on a resource alloca- tion policy trained via reinforcement learning. Our results on SQuAD-Open show that adap- tive computation with global prioritisation im- proves over several strong static and adaptive methods, leading to a 4.3x reduction in compu- tation while retaining 95% performance of the full model. 1 Introduction Open-Domain Question Answering (ODQA) re- quires a system to answer questions using a large collection of documents as the information source. In contrast to context-based machine comprehen- sion, where models are to extract answers from single paragraphs or documents, it poses a funda- mental technical challenge in machine reading at scale (Chen et al., 2017) . Most ODQA systems consist of two-stage pipelines, where 1) a context retriever such as BM25 (Robertson, 2004) or DPR (Karpukhin et al., 2020) first selects a small subset of passages that are likely to contain the answer to the question, and 2) a machine reader such as BERT (Devlin et al., 2019) then examines the retrieved contexts to ex- tract the answer. This two-stage process leads to a computational trade-off that is indicated in Fig. 1. We can run computationally expensive deep net- works on a large number of passages to increase the probability that we find the right answer (“All Lay- ers, All Passages”), or cut the number of passages and layers to reduce the computational footprint at the possible cost of missing an answer (“6 Layers, Top-2 Passages”). We hypothesise that a better accuracy-efficiency trade-off can be found if the computational budget is not allocated statically, but based on the com- plexity of each passage, see “Adaptive Computa- tion” in Fig. 1. If a passage is likely to contain the answer, allocate more computation. If it isn’t, allo- cate less. The idea of conditioning neural network computation based on inputs …

Content truncated to 500 words