An Efficient Memory-Augmented Transformer for Knowledge-Intensive NLP Tasks

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5184 - 5196 December 7-11, 2022 ©2022 Association for Computational Linguistics An Efficient Memory-Augmented Transformer for Knowledge-Intensive NLP Tasks Yuxiang Wu † Yu Zhao ‡ Baotian Hu ‡∗ Pasquale Minervini §† Pontus Stenetorp † Sebastian Riedel † † University College London, London, UK ‡ Harbin Institute of Technology, Shenzhen, PRC § University of Edinburgh, Edinburgh, UK {yuxiang.wu, p.stenetorp, s.riedel}@cs.ucl.ac.uk p.minervini@ed.ac.uk 20s151163@stu.hit.edu.cn hubaotian@hit.edu.cn Abstract Access to external knowledge is essential for many natural language processing tasks, such as question answering and dialogue. Exist- ing methods often rely on a parametric model that stores knowledge in its parameters, or use a retrieval-augmented model that has access to an external knowledge source. Parametric and retrieval-augmented models have comple- mentary strengths in terms of computational efficiency and predictive accuracy. To com- bine the strength of both approaches, we pro- pose the Efficient Memory-Augmented Trans- former (EMAT) – it encodes external knowl- edge into a key-value memory and exploits the fast maximum inner product search for mem- ory querying. We also introduce pre-training tasks that allow EMAT to encode informa- tive key-value representations, and to learn an implicit strategy to integrate multiple mem- ory slots into the transformer. Experiments on various knowledge-intensive tasks such as question answering and dialogue datasets show that, simply augmenting parametric models (T5-base) using our method produces more accurate results (e.g., 25.8 → 44.3 EM on NQ) while retaining a high throughput (e.g., 1000 queries/s on NQ). Compared to retrieval- augmented models, EMAT runs substantially faster across the board and produces more ac- curate results on WoW and ELI5.1 1 Introduction NLP tasks often require knowledge that is not explicitly provided with the input. For example, Open-Domain Question Answering (ODQA) re- quires answering an open-domain question without given context passages (Chen et al., 2017), and like- wise for open-domain dialogue (Dinan et al., 2019). To handle such tasks, one key challenge is storing *Corresponding author. 1Our code and datasets are available at https://github. com/uclnlp/EMAT. and accessing potentially large amounts of knowl- edge. One approach is a parametric method that trains a sequence-to-sequence generator to repre- sent knowledge within model parameters. Petroni et al. (2019) find that pre-trained Language Mod- els (PLMs) learn a partial knowledge base in their parameters, but its coverage is limited. Increasing model size can improve this issue (Raffel et al., 2020; Roberts et al., 2020; Brown et al., 2020); however, larger language models require signifi- cant computational resources. Retrieval-augmented models (Guu et al., 2020; Lewis et al., 2020b; Izacard and Grave, 2021; Das et al., 2022), on the other hand, retrieve relevant passages from an external knowledge source (e.g., Wikipedia), and use the retrieved passages to in- form generation. Despite being more accurate, retrieval-augmented models are often significantly more costly computation-wise than their paramet- ric counterparts, since they require retrieving, en- coding, and integrating the external knowledge at inference time. To combine the strengths of both parametric and retrieval-augmented models, we propose Efficient Memory-Augmented …

Content truncated to 500 words