Rethinking the Value of Transformer Components

Proceedings of the 28th International Conference on Computational Linguistics, pages 6019–6029 Barcelona, Spain (Online), December 8-13, 2020 6019 Rethinking the Value of Transformer Components Wenxuan Wang∗ The Chinese University of Hong Kong jwxwang@gmail.com Zhaopeng Tu† Tencent AI Lab zptu@tencent.com Abstract Transformer becomes the state-of-the-art translation model, while it is not well studied how each intermediate component contributes to the model performance, which poses significant chal- lenges for designing optimal architectures. In this work, we bridge this gap by evaluating the impact of individual component (sub-layer) in trained Transformer models from different per- spectives. Experimental results across language pairs, training strategies, and model capacities show that certain components are consistently more important than the others. We also report a number of interesting findings that might help humans better analyze, understand and improve Transformer models. Based on these observations, we further propose a new training strategy that can improves translation performance by distinguishing the unimportant components in training. 1 Introduction Transformer (Vaswani et al., 2017) has achieved the state-of-the-art performance on a variety of trans- lation tasks. It consists of different stacked components, including self-attention, encoder-attention, and feed-forward layers. However, so far not much is known about the internal properties and functionalities it learns to achieve the performance, which poses significant challenges for designing optimal architec- tures. In this work, we bridge the gap by conducting a granular analysis of components on trained Trans- former models. We attempt to understand how does each component contribute to the model outputs. Specifically, we explore two metrics to evaluate the impact of a particular component on the model per- formance: 1) contribution in information flow that manually masks individual component each time and evaluate the performance without that component; and 2) criticality in representation generalization that depends on how much closer the weights can get for each component to the initial weights while still maintaining performance. Those two metrics evaluate the component importance of a trained Trans- former model from different perspectives. Empirical results on two benchmarking datasets reveal the following observations (§3.1): • The decoder self-attention layers are least important, and the decoder feed-forward layers are most important. • The components that are closer to the model input and output (e.g., lower layers of encoder and higher layers of decoder) are more important than components on other layers. • Upper encoder-attention layers in decoder are more important than lower encoder-attention layers. The findings are consistent across different evaluation metrics, translation datasets, initialization seeds and model capacities, demonstrating their robustness. ∗Work done when interning at Tencent AI Lab. †Zhaopeng Tu is the corresponding author. This work is licensed under a Creative Commons Attribution 4.0 International Licence. Licence details: http:// creativecommons.org/licenses/by/4.0/. 6020 We further analyze the underlying reason (§3.2), and find that lower dropout ratio and more training data lead to less unimportant components. Unimportant components can be identified at early stage of training, which are not due to deficient training. Finally, we show that unimportant components can be rewound (Frankle and Carbin, 2019) to further …

Content truncated to 500 words