The Cascade Transformer: an Application for Efficient Answer Sentence Selection

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5697–5708 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 5697 The Cascade Transformer: an Application for Efficient Answer Sentence Selection Luca Soldaini Amazon Alexa Manhattan Beach, CA, USA lssoldai@amazon.com Alessandro Moschitti Amazon Alexa Manhattan Beach, CA, USA amosch@amazon.com Abstract Large transformer-based language models have been shown to be very effective in many classification tasks. However, their computa- tional complexity prevents their use in appli- cations requiring the classification of a large set of candidates. While previous works have investigated approaches to reduce model size, relatively little attention has been paid to tech- niques to improve batch throughput during in- ference. In this paper, we introduce the Cas- cade Transformer, a simple yet effective tech- nique to adapt transformer-based models into a cascade of rankers. Each ranker is used to prune a subset of candidates in a batch, thus dramatically increasing throughput at in- ference time. Partial encodings from the trans- former model are shared among rerankers, pro- viding further speed-up. When compared to a state-of-the-art transformer model, our ap- proach reduces computation by 37% with al- most no impact on accuracy, as measured on two English Question Answering datasets. 1 Introduction Recent research has shown that transformer-based neural networks can greatly advance the state of the art over many natural language processing tasks. Efforts such as BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019c), XLNet (Dai et al., 2019), and others have led to major advancements in several NLP subfields. These models are able to approximate syntactic and semantic relations be- tween words and their compounds by pre-training on copious amounts of unlabeled data (Clark et al., 2019; Jawahar et al., 2019). Then, they can eas- ily be applied to different tasks by just fine-tuning them on training data from the target domain/task (Liu et al., 2019a; Peters et al., 2019). The im- pressive effectiveness of transformer-based neural networks can be partially attributed to their large number of parameters (ranging from 110 million for “base” models to over 8 billion (Shoeybi et al., 2019)); however, this also makes them rather expen- sive in terms of computation time and resources. Being aware of this problem, the research com- munity has been developing techniques to prune unnecessary network parameters (Lan et al., 2019; Sanh et al., 2019) or optimize the transformer ar- chitecture (Zhang et al., 2018; Xiao et al., 2019). In this paper, we propose a completely differ- ent approach for increasing the efficiency of trans- former models, which is orthogonal to previous work, and thus can be applied in addition to any of the methods described above. Its main idea is that a large class of NLP problems requires choos- ing one correct candidate among many. For some applications, this often entails running the model over hundreds or thousands of instances. However, it is well-known that, in many cases, some candi- dates can be more easily excluded from the optimal solution (Land and …

Content truncated to 500 words