What Works and Doesn’t Work, A Deep Decoder for Neural Machine Translation

Findings of the Association for Computational Linguistics: ACL 2022, pages 459 - 471 May 22-27, 2022 c⃝2022 Association for Computational Linguistics What Works and Doesn’t Work, A Deep Decoder for Neural Machine Translation Zuchao Li1, Yiran Wang2, Masao Utiyama2,∗, Eiichiro Sumita2, Hai Zhao1∗, and Taro Watanabe3 1Shanghai Jiao Tong University (SJTU), Shanghai, China 2National Institute of Information and Communications Technology (NICT), Kyoto, Japan 3Nara Institute of Science and Technology (NAIST), Nara, Japan charlee@sjtu.edu.cn, {yiran.wang,mutiyama}@nict.go.jp, eiichiro.sumita@nict.go.jp, zhaohai@cs.sjtu.edu.cn, taro@is.naist.jp Abstract Deep learning has demonstrated performance advantages in a wide range of natural language processing tasks, including neural machine translation (NMT). Transformer NMT models are typically strengthened by deeper encoder layers, but deepening their decoder layers usu- ally results in failure. In this paper, we first identify the cause of the failure of the deep de- coder in the Transformer model. Inspired by this discovery, we then propose approaches to improving it, with respect to model structure and model training, to make the deep decoder practical in NMT. Specifically, with respect to model structure, we propose a cross-attention drop mechanism to allow the decoder layers to perform their own different roles, to reduce the difficulty of deep-decoder learning. For model training, we propose a collapse reducing train- ing approach to improve the stability and effec- tiveness of deep-decoder training. We experi- mentally evaluated our proposed Transformer NMT model structure modification and novel training methods on several popular machine translation benchmarks. The results showed that deepening the NMT model by increasing the number of decoder layers successfully pre- vented the deepened decoder from degrading to an unconditional language model. In contrast to prior work on deepening an NMT model on the encoder, our method can deepen the model on both the encoder and decoder at the same time, resulting in a deeper model and improved performance. 1 Introduction With the help of the deep neural network, the feature extraction capability of models has been ∗Corresponding author. Zuchao Li, and Hai Zhao are with the Department of Computer Science and Engineering, and with Key Laboratory of Shanghai Education Commission for Intelligent Interaction and Cognitive Engineering, Shanghai Jiao Tong University. This paper was finished when Zuchao Li was a fixed term technical researcher at NICT. This work was partially funded by the Key Projects of National Natural Science Foundation of China (U1836222 and 61733011). substantially enhanced (Schmidhuber, 2015; Le- Cun et al., 2015). Deep neural network models are also popular for natural language processing (NLP) tasks. The most typical deep neural net- work model in NLP is based on the convolutional neural network (CNN) (Gehring et al., 2017) and Transformer (Vaswani et al., 2017) structures, and the deep pretrained Transformer language model has begun to dominate NLP. The deep neural net- work model has also attracted substantial interest in neural machine translation (NMT), for both theo- retical research (Wang et al., 2019; Li et al., 2020a, 2021a; Kong et al., 2021) and competition evalua- tion (Zhang et al., 2020; Wu et al., 2020b,a; Meng et al., …

Content truncated to 500 words