Decoupled Transformer for Scalable Inference in Open-domain Question Answering

Proceedings of Recent Advances in Natural Language Processing, pages 386–393 Sep 1–3, 2021. https://doi.org/10.26615/978-954-452-072-4_044 386 Decoupled Transformer for Scalable Inference in Open-domain Question Answering Haytham ElFadeel haythamf@fb.com Stan Peshterliev stanvp@fb.com Abstract Large transformer models, such as BERT, achieve state-of-the-art results in machine read- ing comprehension (MRC) for open-domain question answering (QA). However, transform- ers have a high computational cost for infer- ence which makes them hard to apply to online QA systems for applications like voice assis- tants. To reduce computational cost and la- tency, we propose decoupling the transformer MRC model into input-component and cross- component. The decoupling allows for part of the representation computation to be performed offline and cached for online use. To retain the decoupled transformer accuracy, we devised a knowledge distillation objective from a stan- dard transformer model. Moreover, we intro- duce learned representation compression layers which help reduce by four times the storage re- quirement for the cache. In experiments on the SQUAD 2.0 dataset, a decoupled transformer reduces the computational cost and latency of open-domain MRC by 30-40% with only 1.2 points worse F1-score compared to a standard transformer. 1 Introduction Open-domain question answering (QA) aims to an- swer questions from a collection of text passages. It is an important and challenging task with appli- cation to several domains such as web search and voice assistants. The most popular architecture for open-domain QA is retriever-reader (Chen et al., 2017). Given a question, the retriever uses an in- formation retrieval (IR) system over a collection of passages to return top-K results that are most likely to contain an answer. Then, the reader uses a machine reading comprehension (MRC) model on each of the top-K results to find an answer. In the end, the top-K MRC answers are ranked to produce a final answer. For both the retriever and the reader, large trans- former models such as BERT (Devlin et al., 2018), RoBERTa (Liu et al., 2019) and ELECTRA (Clark et al., 2020) achieve state-of-the-art results. disadvantage of large transformer models is theA high computational cost for inference which makes them hard to apply to online runtime systems, e.g. voice-assistants. Transformers’ computational cost comes from three major factors. Firstly, the size of the feed-forward layers which project to an inter- mediate higher dimension and projects back to the original dimension. Secondly, the multi-head self- attention has quadratic computational complexity in the sequence length. Thirdly, the total number of layers. Dense passage retrieval (DPR) (Karpukhin et al., 2020) is a retriever model which uses a transformer question encoder and transformer passage encoder to capture semantic similarity. The question and passage encoders are trained such that passages that are likely to contain an answer have a large em- bedding dot product with the question embedding. The embeddings for the passages are generated offline and indexed for efficient distributed KNN search (Johnson et al., 2017), and only the embed- ding for the question is generated at runtime. Since questions are usually short, the retriever runtime inference computational cost …

Content truncated to 500 words