Towards Linear Time Neural Machine Translation with Capsule Networks

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 803–812, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 803 Towards Linear Time Neural Machine Translation with Capsule Networks Mingxuan Wang1 Jun xie2 Zhixing Tan2 Jinsong Su3 Deyi Xiong4 Lei Li1 1ByteDance AI Lab, Beijing, China {wangmingxuan.89,lilei.lab}@bytedance.com 2Mobile Internet Group, Tencent Technology Co., Ltd 3Xiamen University, Xiamen, China 4Tianjin University, Tianjin, China Abstract In this study, we first investigate a novel cap- sule network with dynamic routing for linear time Neural Machine Translation (NMT), re- ferred as CAPSNMT. CAPSNMT uses an ag- gregation mechanism to map the source sen- tence into a matrix with pre-determined size, and then applys a deep LSTM network to decode the target sequence from the source representation. Different from the previous work (Sutskever et al., 2014) to store the source sentence with a passive and bottom- up way, the dynamic routing policy encodes the source sentence with an iterative process to decide the credit attribution between nodes from lower and higher layers. CAPSNMT has two core properties: it runs in time that is linear in the length of the sequences and provides a more flexible way to aggregate the part-whole information of the source sen- tence. On WMT14 English-German task and a larger WMT14 English-French task, CAP- SNMT achieves comparable results with the Transformer system. To the best of our knowl- edge, this is the first work that capsule net- works have been empirically investigated for sequence to sequence problems.1 1 Introduction Neural Machine Translation (NMT) is an end- to-end learning approach to machine translation which has recently shown promising results on multiple language pairs (Luong et al., 2015; Shen et al., 2015; Wu et al., 2016; Gehring et al., 2017a; Kalchbrenner et al., 2016; Sennrich et al., 2015; Vaswani et al., 2017). Unlike conventional Statis- tical Machine Translation (SMT) systems (Koehn et al., 2003; Chiang, 2005) which consist of mul- tiple separately tuned components, NMT aims at building upon a single and large neural network 1The work is partially done when the first author worked at Tencent. to directly map input text to associated output text (Sutskever et al., 2014). In general, there are several research lines of NMT architectures, among which the Enc-Dec NMT (Sutskever et al., 2014) and the Enc-Dec Att NMT are of typical representation (Bahdanau et al., 2014; Wu et al., 2016; Vaswani et al., 2017). The Enc-Dec represents the source inputs with a fixed dimensional vector and the target sequence is generated from this vector word by word. The Enc-Dec, however, does not preserve the source sequence resolution, a feature which aggravates learning for long sequences. This results in the computational complexity of decoding process be- ing O(|S| + |T|), with |S| denoting the source sentence length and |T| denoting the target sen- tence length. The Enc-Dec Att preserves the reso- lution of the source sentence which frees the neu- ral model from …

Content truncated to 500 words