Improving Transformer Models by Reordering their Sublayers

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2996–3005 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 2996 Improving Transformer Models by Reordering their Sublayers Ofir Press♦ Noah A. Smith♦♠ Omer Levy♣ ♦Paul G. Allen School of Computer Science & Engineering, University of Washington ♠Allen Institute for AI ♣Facebook AI Research Abstract Multilayer transformer networks consist of in- terleaved self-attention and feedforward sub- layers. Could ordering the sublayers in a dif- ferent pattern lead to better performance? We generate randomly ordered transformers and train them with the language modeling objec- tive. We observe that some of these models are able to achieve better performance than the interleaved baseline, and that those successful variants tend to have more self-attention at the bottom and more feedforward sublayers at the top. We propose a new transformer pattern that adheres to this property, the sandwich trans- former, and show that it improves perplexity on multiple word-level and character-level lan- guage modeling benchmarks, at no cost in pa- rameters, memory, or training time. However, the sandwich reordering pattern does not guar- antee performance gains across every task, as we demonstrate on machine translation mod- els. Instead, we suggest that further explo- ration of task-specific sublayer reorderings is needed in order to unlock additional gains.1 1 Introduction The transformer layer (Vaswani et al., 2017) is cur- rently the primary modeling component in natural language processing, playing a lead role in recent innovations such as BERT (Devlin et al., 2019) and GPT-2 (Radford et al., 2019). Each transformer layer consists of a self-attention sublayer (s) fol- lowed by a feedforward sublayer (f), creating an in- terleaving pattern of self-attention and feedforward sublayers (sfsfsf · · · ) throughout a multilayer transformer model. To the best of our knowledge, there is no reason to expect this particular pattern to be optimal. We conduct a series of explorations to obtain insights about the nature of transformer orderings that work well, and based on this, we 1Our code is available at https://github.com/ ofirpress/sandwich_transformer sfsfsfsfsfsfsfsfsfsfsfsfsfsf (a) Interleaved Transformer sssssssfsfsfsfsfsfsfsfffffff (b) Sandwich Transformer Figure 1: A transformer model (a) is composed of inter- leaved self-attention (green) and feedforward (purple) sublayers. Our sandwich transformer (b), a reordering of the transformer sublayers, performs better on lan- guage modeling. Input flows from left to right. design a new transformer ordering pattern that im- proves upon the baseline. First, we generate random transformer models, varying the number of each type of sublayer, and their ordering, while keeping the number of pa- rameters constant. We train these models on the standard WikiText-103 word-level language mod- eling benchmark (Merity et al., 2016), and observe that some of these random models outperform the original interleaved transformer model, even when the number of self-attention and feedforward layers is not equal. Our analysis shows that models with more self-attention toward the bottom and more feedforward sublayers toward the top tend to per- form better in general. Based on this insight, we design a new …

Content truncated to 500 words