Deep Neural Machine Translation with Linear Associative Unit

Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 136–145 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1013 Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 136–145 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1013 Deep Neural Machine Translation with Linear Associative Unit Mingxuan Wang1 Zhengdong Lu2 Jie Zhou2 Qun Liu4,5 1Mobile Internet Group, Tencent Technology Co., Ltd wangmingxuan@ict.ac.cn 2DeeplyCurious.ai 3 Insititute of Deep Learning Research, Baidu Co., Ltd 4 Institute of Computing Technology, Chinese Academy of Sciences 5ADAPT Centre, School of Computing, Dublin City University Abstract Deep Neural Networks (DNNs) have provably enhanced the state-of-the-art Neural Machine Translation (NMT) with their capability in modeling com- plex functions and capturing com- plex linguistic structures. However NMT systems with deep architecture in their encoder or decoder RNNs of- ten suffer from severe gradient diffu- sion due to the non-linear recurrent ac- tivations, which often make the opti- mization much more difficult. To ad- dress this problem we propose novel linear associative units (LAU) to re- duce the gradient propagation length inside the recurrent unit. Different from conventional approaches (LSTM unit and GRU), LAUs utilizes lin- ear associative connections between in- put and output of the recurrent unit, which allows unimpeded information flow through both space and time di- rection. The model is quite simple, but it is surprisingly effective. Our empirical study on Chinese-English translation shows that our model with proper configuration can improve by 11.7 BLEU upon Groundhog and the best reported results in the same set- ting. On WMT14 English-German task and a larger WMT14 English-French task, our model achieves comparable results with the state-of-the-art. 1 Introduction Neural Machine Translation (NMT) is an end- to-end learning approach to machine transla- tion which has recently shown promising re- sults on multiple language pairs (Luong et al., 2015; Shen et al., 2015; Wu et al., 2016; Zhang et al., 2016; Tu et al., 2016; Zhang and Zong, 2016; Jean et al., 2015; Meng et al., 2015). Un- like conventional Statistical Machine Transla- tion (SMT) systems (Koehn et al., 2003; Chi- ang, 2005; Liu et al., 2006; Xiong et al., 2006; Mi et al., 2008) which consist of multiple sep- arately tuned components, NMT aims at build- ing upon a single and large neural network to directly map input text to associated output text. Typical NMT models consists of two re- current neural networks (RNNs), an encoder to read and encode the input text into a distributed representation and a decoder to generate trans- lated text conditioned on the input representa- tion (Sutskever et al., 2014; Bahdanau et al., 2014). Driven by the breakthrough achieved in computer vision (He et al., 2015; Srivastava et al., 2015), research in NMT has recently turned towards studying Deep Neural Net- works (DNNs). Wu et al. (2016) and Zhou et al. (2016) found that deep architectures in both the encoder and decoder are essential for …

Content truncated to 500 words