schuBERT: Optimizing Elements of BERT

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2807–2818 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 2807 schuBERT: Optimizing Elements of BERT Ashish Khetan Amazon AWS New York, USA khetan@amazon.com Zohar Karnin Amazon AWS New York, USA zkarnin@amazon.com Abstract Transformers (Vaswani et al., 2017) have grad- ually become a key component for many state-of-the-art natural language representa- tion models. A recent Transformer based model- BERT (Devlin et al., 2018) achieved state-of-the-art results on various natural lan- guage processing tasks, including GLUE, SQuAD v1.1, and SQuAD v2.0. This model however is computationally prohibitive and has a huge number of parameters. In this work we revisit the architecture choices of BERT in efforts to obtain a lighter model. We focus on reducing the number of parameters yet our methods can be applied towards other objec- tives such FLOPs or latency. We show that much efficient light BERT models can be ob- tained by reducing algorithmically chosen cor- rect architecture design dimensions rather than reducing the number of Transformer encoder layers. In particular, our schuBERT gives 6.6% higher average accuracy on GLUE and SQuAD datasets as compared to BERT with three encoder layers while having the same number of parameters. 1 Introduction Transformer (Vaswani et al., 2017) based mod- els have achieved state-of-the-art performance for many natural language processing tasks (Dai and Le, 2015; Peters et al., 2018; Radford et al., 2018; Howard and Ruder, 2018). These include machine translation (Vaswani et al., 2017; Ott et al., 2018), question-answering tasks (Devlin et al., 2018), nat- ural language inference (Bowman et al., 2015; Williams et al., 2017) and semantic role labeling (Strubell et al., 2018). A recent Transformer based model BERT (De- vlin et al., 2018) achieved state-of-the-art results on various natural language processing tasks including GLUE, SQuAD v1.1 and SQuAD v2.0. BERT’s model architecture is a multi-layer bidirectional Transformer encoder based on the original imple- mentation described in Vaswani et al. (2017). Following the seminal results obtained by the BERT model, several follow up studies explored methods for improving them further. XLNet (Yang et al., 2019) adds autoregressive capabilities to BERT, improving its quality, though at the cost of additional compute requirements. RoBERTa (Liu et al., 2019) modifies the training procedure of BERT and provides pre-training methods that sig- nificantly improve its performance. Two notable pa- pers exploring the architecture design of the BERT are following. Michel et al. (2019) examines the importance of attention heads in BERT architec- ture, highlighting scenarios where attention heads may be pruned. The main objective of the paper is to provide techniques for pruning attention head, and as such the amount of experiments performed on BERT is limited to a single task (MNLI). AL- BERT (Lan et al., 2019) proposes two methods for reducing the number of parameters in BERT. The first is via parameter sharing across layers, and the second is by factorizing the embedding layers. We note (this was mentioned in the conclusion section of the paper) …

Content truncated to 500 words