Chunk-based Decoder for Neural Machine Translation

Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 1901–1912 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1174 Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 1901–1912 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1174 Chunk-based Decoder for Neural Machine Translation Shonosuke Ishiwatar†∗ Jingtao Yao‡∗ Shujie Liu§ Mu Li§ Ming Zhou§ Naoki Yoshinaga¶ Masaru Kitsuregawa∥¶ Weijia Jia‡ † The University of Tokyo ‡ Shanghai Jiao Tong University § Microsoft Research Asia ¶ Institute of Industrial Science, the University of Tokyo ∥ National Institute of Informatics †¶∥{ishiwatari, ynaga, kitsure}@tkl.iis.u-tokyo.ac.jp ‡{yjt1995@, jia-wj@cs.}sjtu.edu.cn §{shujliu, muli, mingzhou}@microsoft.com Abstract Chunks (or phrases) once played a piv- otal role in machine translation. By us- ing a chunk rather than a word as the basic translation unit, local (intra- chunk) and global (inter-chunk) word or- ders and dependencies can be easily mod- eled. The chunk structure, despite its im- portance, has not been considered in the decoders used for neural machine trans- lation (NMT). In this paper, we propose chunk-based decoders for NMT, each of which consists of a chunk-level decoder and a word-level decoder. The chunk- level decoder models global dependencies while the word-level decoder decides the local word order in a chunk. To output a target sentence, the chunk-level decoder generates a chunk representation contain- ing global information, which the word- level decoder then uses as a basis to pre- dict the words inside the chunk. Experi- mental results show that our proposed de- coders can significantly improve transla- tion performance in a WAT ’16 English- to-Japanese translation task. 1 Introduction Neural machine translation (NMT) performs end- to-end translation based on a simple encoder- decoder model (Kalchbrenner and Blunsom, 2013; Sutskever et al., 2014; Cho et al., 2014b) and has now overtaken the classical, complex statis- tical machine translation (SMT) in terms of perfor- mance and simplicity (Sennrich et al., 2016; Lu- ong and Manning, 2016; Cromieres et al., 2016; Neubig, 2016). In NMT, an encoder first maps a source sequence into vector representations and ∗Contribution during internship at Microsoft Research. !"# $ %&"'()**'+ ,+**-+.(*(&/01(/2 3&# !" # $ %& ' ()* +* , -& + . 4((0 -+.( (&/05 ,+*6&78 Figure 1: Translation from English to Japanese. The function words are underlined. a decoder then maps the vectors into a target se- quence (§ 2). This simple framework allows re- searchers to incorporate the structure of the source sentence as in SMT by leveraging various architec- tures as the encoder (Kalchbrenner and Blunsom, 2013; Sutskever et al., 2014; Cho et al., 2014b; Eriguchi et al., 2016b). Most of the NMT models, however, still rely on a sequential decoder based on a recurrent neural network (RNN) due to the difficulty in capturing the structure of a target sen- tence that is unseen during translation. With the sequential decoder, however, there are two problems to be solved. First, it is difficult to model long-distance dependencies …

Content truncated to 500 words