Dense Information Flow for Neural Machine Translation

Proceedings of NAACL-HLT 2018, pages 1294–1303 New Orleans, Louisiana, June 1 - 6, 2018. c⃝2018 Association for Computational Linguistics Dense Information Flow for Neural Machine Translation Yanyao Shen1, Xu Tan2, Di He3, Tao Qin2, and Tie-Yan Liu2 1University of Texas at Austin 2Microsoft Research, Asia 3Key Laboratory of Machine Perception, MOE, School of EECS, Peking University shenyanyao@utexas.edu, {xuta,taoqin,tie-yan.liu}@microsoft.com,di he@pku.edu.cn Abstract Recently, neural machine translation has achieved remarkable progress by intro- ducing well-designed deep neural net- works into its encoder-decoder frame- work. From the optimization perspective, residual connections are adopted to im- prove learning performance for both en- coder and decoder in most of these deep architectures, and advanced attention con- nections are applied as well. Inspired by the success of the DenseNet model in computer vision problems, in this pa- per, we propose a densely connected NMT architecture (DenseNMT) that is able to train more efficiently for NMT. The pro- posed DenseNMT not only allows dense connection in creating new features for both encoder and decoder, but also uses the dense attention structure to improve at- tention quality. Our experiments on mul- tiple datasets show that DenseNMT struc- ture is more competitive and efficient. 1 Introduction Neural machine translation (NMT) is a challeng- ing task that attracts lots of attention in recent years. Starting from the encoder-decoder frame- work (Cho et al., 2014), NMT starts to show promising results in many language pairs. The evolving structures of NMT models in recent years have made them achieve higher scores and become more favorable. The attention mecha- nism (Bahdanau et al., 2015) added on top of encoder-decoder framework is shown to be very useful to automatically find alignment structure, and single-layer RNN-based structure has evolved into deeper models with more efficient transfor- mation functions (Gehring et al., 2017; Kaiser et al., 2017; Vaswani et al., 2017). One major challenge of NMT is that its models are hard to train in general due to the complex- ity of both the deep models and languages. From the optimization perspective, deeper models are hard to efficiently back-propagate the gradients, and this phenomenon as well as its solution is bet- ter explored in the computer vision society. Resid- ual networks (ResNet) (He et al., 2016) achieve great performance in a wide range of tasks, in- cluding image classification and image segmen- tation. Residual connections allow features from previous layers to be accumulated to the next layer easily, and make the optimization of the model ef- ficiently focus on refining upper layer features. NMT is considered as a challenging problem due to its sequence-to-sequence generation frame- work, and the goal of comprehension and reor- ganizing from one language to the other. Apart from the encoder block that works as a feature generator, the decoder network combining with the attention mechanism bring new challenges to the optimization of the models. While nowadays best-performing NMT systems use residual con- nections, we question whether this is the most ef- ficient way to propagate information through deep models. In this paper, …

Content truncated to 500 words