Depth Growing for Neural Machine Translation

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5558–5563 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 5558 Depth Growing for Neural Machine Translation Lijun Wu1,∗, Yiren Wang2,∗, Yingce Xia3,†, Fei Tian3, Fei Gao3, Tao Qin3, Jianhuang Lai1, Tie-Yan Liu3 1School of Data and Computer Science, Sun Yat-sen University; 2 University of Illinois at Urbana-Champaign; 3 Microsoft Research Asia 1{wulijun3, stsljh}@mail2.sysu.edu.cn, 2yiren@illinois.edu, 3{Yingce.Xia, fetia, feiga, taoqin, tyliu}@microsoft.com Abstract While very deep neural networks have shown effectiveness for computer vision and text classification applications, how to increase the network depth of neural machine translation (NMT) models for better translation quality re- mains a challenging problem. Directly stack- ing more blocks to the NMT model results in no improvement and even reduces perfor- mance. In this work, we propose an effec- tive two-stage approach with three specially designed components to construct deeper NMT models, which result in significant improvements over the strong Transformer baselines on WMT14 English→German and English→French translation tasks. 1 Introduction Neural machine translation (briefly, NMT), which is built upon deep neural networks, has gained rapid progress in recent years (Bahdanau et al., 2014; Sutskever et al., 2014; Sennrich et al., 2015; He et al., 2016a; Sennrich et al., 2016a; Xia et al., 2017; Wang et al., 2019) and achieved signifi- cant improvement in translation quality (Hassan et al., 2018). Variants of network structures have been applied in NMT such as LSTM (Wu et al., 2016), CNN (Gehring et al., 2017) and Trans- former (Vaswani et al., 2017). Training deep networks has always been a chal- lenging problem, mainly due to the difficulties in optimization for deep architecture. Breakthroughs have been made in computer vision to enable deeper model construction via advanced initializa- tion schemes (He et al., 2015), multi-stage train- ing strategy (Simonyan and Zisserman, 2014), and novel model architectures (Srivastava et al., 2015; He et al., 2016b). While constructing very deep ∗The first two authors contributed equally to this work. This work is conducted at Microsoft Research Asia. †Corresponding author. Figure 1: Performances of Transformer models with different number of encoder/decoder blocks (recorded on x-axis) on WMT14 En→De translation task. † de- notes the result reported in (Vaswani et al., 2017). neural networks with tens and even more than a hundred blocks have shown effectiveness in im- age recognition (He et al., 2016b), question an- swering and text classification (Devlin et al., 2018; Radford et al., 2019), scaling up model capac- ity with very deep network remains challenging for NMT. The NMT models are generally con- structed with up to 6 encoder and decoder blocks in both state-of-the-art research work and cham- pion systems of machine translation competition. For example, the LSTM-based models are usually stacked for 4 (Stahlberg et al., 2018) or 6 (Chen et al., 2018) blocks, and the state-of-the-art Trans- former models are equipped with a 6-block en- coder and decoder (Vaswani et al., 2017; Junczys- Dowmunt, 2018; Edunov et al., 2018). Increasing the NMT …

Content truncated to 500 words