DeFormer: Decomposing Pre-trained Transformers for Faster Question Answering

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4487–4497 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 4487 DeFormer: Decomposing Pre-trained Transformers for Faster Question Answering Qingqing Cao, Harsh Trivedi, Aruna Balasubramanian, Niranjan Balasubramanian Department of Computer Science Stony Brook University Stony Brook, NY 11794, USA {qicao,hjtrivedi,arunab,niranjan}@cs.stonybrook.edu Abstract Transformer-based QA models use input-wide self-attention – i.e. across both the question and the input passage – at all layers, causing them to be slow and memory-intensive. turns out that we can get by without input-It wide self-attention at all layers, especially in the lower layers. We introduce DeFormer, a decomposed transformer, which substitutes the full self-attention with question-wide and passage-wide self-attentions in the lower lay- ers. This allows for question-independent pro- cessing of the input text representations, which in turn enables pre-computing passage rep- resentations reducing runtime compute dras- tically. Furthermore, because DeFormer is largely similar to the original model, we can initialize DeFormer with the pre-training weights of a standard transformer, and directly fine-tune on the target QA dataset. We show DeFormer versions of BERT and XLNet can be used to speed up QA by over 4.3x and with simple distillation-based losses they in- cur only a 1% drop in accuracy. We open source the code at https://github.com/ StonyBrookNLP/deformer. 1 Introduction There is an increasing need to push question an- swering (QA) models in large volume web scale services (Google, 2019) and also to push them to re- source constrained mobile devices for privacy and other performance reasons (Cao et al., 2019). State- of-the-art QA systems, like many other NLP appli- cations, are built using large pre-trained Transform- ers (e.g., BERT (Devlin et al., 2019), XLNet (Yang et al., 2019), Roberta (Liu et al., 2019)). However, inference in these models requires prohibitively high-levels of runtime compute and memory mak- ing it expensive to support large volume deploy- ments in data centers and infeasible to run on re- source constrained mobile devices. Our goal is to take pre-trained Transformer- based models and modify them to enable faster Decompose CLS My name SEP Your ? CLS My name SEP Your ? Transformer DeFormer Figure 1: Original Transformer applies full self- attention to encode the concatenated question and pas- sage sequence, while DeFormer encodes the question and passage independently in the lower layers and pro- cesses them jointly in the higher layers. inference for QA without having to repeat the pre- training. This is a critical requirement if we want to explore many points in the accuracy versus speed trade-off because pre-training is expensive. The main compute bottleneck in Transformer- based models is the input-wide self-attention com- putation at each layer. In reading comprehension style QA, this amounts to computing self-attention over the question and the context text together. This helps the models create highly effective question- dependent context representations and vice-versa. Of these, building representations of the context takes more time because it is typically much longer than the question. If the context can be …

Content truncated to 500 words