A Character-level Decoder without Explicit Segmentation for Neural Machine Translation

Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 1693–1703, Berlin, Germany, August 7-12, 2016. c⃝2016 Association for Computational Linguistics A Character-Level Decoder without Explicit Segmentation for Neural Machine Translation Junyoung Chung Universit´e de Montr´eal junyoung.chung@umontreal.ca Kyunghyun Cho New York University Yoshua Bengio Universit´e de Montr´eal CIFAR Senior Fellow Abstract The existing machine translation systems, whether phrase-based or neural, have relied almost exclusively on word-level modelling with explicit segmentation. In this paper, we ask a fundamental question: can neural machine translation generate a character sequence without any explicit segmentation? To answer this question, we evaluate an attention-based encoder– decoder with a subword-level encoder and a character-level decoder on four language pairs–En-Cs, En-De, En-Ru and En-Fi– using the parallel corpora from WMT’15. Our experiments show that the models with a character-level decoder outperform the ones with a subword-level decoder on all of the four language pairs. Further- more, the ensembles of neural models with a character-level decoder outperform the state-of-the-art non-neural machine trans- lation systems on En-Cs, En-De and En-Fi and perform comparably on En-Ru. 1 Introduction The existing machine translation systems have re- lied almost exclusively on word-level modelling with explicit segmentation. This is mainly due to the issue of data sparsity which becomes much more severe, especially for n-grams, when a sen- tence is represented as a sequence of characters rather than words, as the length of the sequence grows significantly. In addition to data sparsity, we often have a priori belief that a word, or its segmented-out lexeme, is a basic unit of meaning, making it natural to approach translation as map- ping from a sequence of source-language words to a sequence of target-language words. This has continued with the more recently proposed paradigm of neural machine transla- tion, although neural networks do not suffer from character-level modelling and rather suffer from the issues specific to word-level modelling, such as the increased computational complexity from a very large target vocabulary (Jean et al., 2015; Lu- ong et al., 2015b). Therefore, in this paper, we ad- dress a question of whether neural machine trans- lation can be done directly on a sequence of char- acters without any explicit word segmentation. To answer this question, we focus on represent- ing the target side as a character sequence. We evaluate neural machine translation models with a character-level decoder on four language pairs from WMT’15 to make our evaluation as convinc- ing as possible. We represent the source side as a sequence of subwords extracted using byte-pair encoding from Sennrich et al. (2015), and vary the target side to be either a sequence of subwords or characters. On the target side, we further design a novel recurrent neural network (RNN), called bi- scale recurrent network, that better handles multi- ple timescales in a sequence, and test it in addition to a naive, stacked recurrent neural network. On all of the four language pairs–En-Cs, En-De, En-Ru and En-Fi–, the models with a character- level decoder outperformed the ones with …

Content truncated to 500 words