Length-Adaptive Transformer: Train Once with Length Drop, Use Anytime with Search

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 6501–6511 August 1–6, 2021. ©2021 Association for Computational Linguistics 6501 Length-Adaptive Transformer: Train Once with Length Drop, Use Anytime with Search Gyuwan Kim Clova AI, NAVER Corp. gyuwan.kim@navercorp.com Kyunghyun Cho New York University kyunghyun.cho@nyu.edu Abstract Despite transformers’ impressive accuracy, their computational cost is often prohibitive to use with limited computational resources. Most previous approaches to improve infer- ence efficiency require a separate model for each possible computational budget. In this paper, we extend PoWER-BERT (Goyal et al., 2020) and propose Length-Adaptive Trans- former that can be used for various infer- ence scenarios after one-shot training. We train a transformer with LengthDrop, a struc- tural variant of dropout, which stochastically determines a sequence length at each layer. We then conduct a multi-objective evolution- ary search to find a length configuration that maximizes the accuracy and minimizes the efficiency metric under any given computa- tional budget. Additionally, we significantly extend the applicability of PoWER-BERT be- yond sequence-level classification into token- level classification with Drop-and-Restore pro- cess that drops word-vectors temporarily in in- termediate layers and restores at the last layer if necessary. We empirically verify the util- ity of the proposed approach by demonstrating the superior accuracy-efficiency trade-off un- der various setups, including span-based ques- tion answering and text classification. Code is available at https://github.com/clovaai/length- adaptive-transformer. 1 Introduction Pre-trained language models (Peters et al., 2018; Devlin et al., 2018; Radford et al., 2019; Yang et al., 2019; He et al., 2020) have achieved notable improvements in various natural language process- ing (NLP) tasks. Most of them rely on transform- ers (Vaswani et al., 2017), and the number of model parameters ranges from hundreds of millions to bil- lions (Shoeybi et al., 2019; Raffel et al., 2019; Ka- plan et al., 2020; Brown et al., 2020). Despite this high accuracy, excessive computational overhead during inference, both in terms of time and memory, has hindered its use in real applications. This level of excessive computation has further raised the con- cern over energy consumption as well (Schwartz et al., 2019; Strubell et al., 2019; Cao et al., 2020). Recent studies have attempted to address these concerns regarding large-scale transformers’ com- putational and energy efficiency (see §7 for a more extensive discussion.) Among these, we focus on PoWER-BERT (Goyal et al., 2020) which pro- gressively reduces sequence length by eliminat- ing word-vectors based on the attention values as passing layers. PoWER-BERT establishes the su- periority of accuracy-time trade-off over earlier approaches (Sanh et al., 2019; Sun et al., 2019; Michel et al., 2019). However, it requires us to train a separate model for each efficiency constraint. In this paper, we thus develop a framework based on PoWER-BERT such that we can train a single model that can be adapted in the inference time to meet any given efficiency target. In order to train a transformer to cope with a di- verse set …

Content truncated to 500 words