Do Transformer Modifications Transfer Across Implementations and Applications?

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5758–5773 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 5758 Do Transformer Modifications Transfer Across Implementations and Applications? Sharan Narang∗ Hyung Won Chung Yi Tay William Fedus Thibault Fevry† Michael Matena† Karishma Malkan† Noah Fiedel Noam Shazeer Zhenzhong Lan† Yanqi Zhou Wei Li Nan Ding Jake Marcus Adam Roberts Colin Raffel† Abstract The research community has proposed co- pious modifications to the Transformer ar- chitecture since it was introduced over three years ago, relatively few of which have seen widespread adoption. In this paper, we comprehensively evaluate many of these modifications in a shared exper- imental setting that covers most of the common uses of the Transformer in natu- ral language processing. Surprisingly, we find that most modifications do not mean- ingfully improve performance. Further- more, most of the Transformer variants we found beneficial were either developed in the same codebase that we used or are rel- atively minor changes. We conjecture that performance improvements may strongly depend on implementation details and cor- respondingly make some recommendations for improving the generality of experimen- tal results. 1 Introduction Much of the empirical success of deep learn- ing can be attributed to advances in meth- ods for building and training neural net- works. These advances include improved op- timizers (Sutskever et al., 2013; Hinton et al., 2012; Kingma and Ba, 2014; Shazeer and Stern, 2018a), regularization schemes (Srivas- tava et al., 2014; Zhang et al., 2017; Neelakan- tan et al., 2015), and model architectures (He et al., 2016; Hochreiter and Schmidhuber, 1997; Vaswani et al., 2017). An aspiration underly- ing much of this work is that an improvement to a particular machine learning pipeline will yield equal-or-better performance on any task that the pipeline is applicable to. For example, residual connections in convolutional networks (He et al., 2016) are designed to ideally improve ∗Correspondence to sharannarang@google.com †Work completed while at Google performance on any task where these models are applicable (image classification, semantic segmentation, etc.). In practice, when propos- ing a new improvement, it is impossible to test it on every applicable downstream task, so re- searchers must select a few representative tasks to evaluate it on. However, the proposals that are ultimately adopted by the research com- munity and practitioners tend to be those that reliably improve performance across a wide variety of tasks “in the wild”. The Transformer architecture (Vaswani et al., 2017) is an example of a seminal im- provement in the field of deep learning. Cur- rently, the Transformer is the de facto archi- tecture of choice for processing sequential data and is starting to be applied to vision appli- cations (e.g. Dosovitskiy et al. (2020)). Since being introduced three years ago, many modi- fications to the Transformer architecture have been proposed. However, the most widely-used applications of the Transformer architecture (e.g. Devlin et al. (2018); Yang et al. (2019); Radford et al. (2018); Raffel et al. (2019)) incor- porate few of these modifications. Instead, …

Content truncated to 500 words