Improving Neural Machine Translation on resource-limited pairs using auxiliary data of a third language

Improving Neural Machine Translation on resource-limited pairs using auxiliary data of a third language Ander Mart´ınez ander.martinez.zy4@is.naist.jp Yuji Matsumoto matsu@is.naist.jp Graduate School of Information Science, Nara Institute of Science and Technology 8916-5 Takayama, Ikoma, Nara, 630-0192, Japan Abstract In the recent years interest in Deep Neural Networks (DNN) has grown in the field of Natural Language Processing, as new training methods have been proposed. The usage of DNN has achieved state-of-the-art performance in various areas. Neural Machine Translation (NMT) described by Bahdanau et al. (2014) and its successive variations have shown promising results. DNN, however, tend to over-fit on small data-sets, which makes this method impracticable for resource-limited language pairs. This article combines three different ideas (splitting words into smaller units, using an extra dataset of a related language pair and using monolingual data) for improving the performance of NMT models on language pairs with limited data. Our experiments show that, in some cases, our proposed approach to subword-units performs better than BPE (Byte pair encoding) and that auxiliary language-pairs and monolingual data can help improve the performance of languages with limited resources. 1 Introduction In the recent years interest in Deep Neural Networks (DNN) has grown in the field of Natu- ral Language Processing (NLP), as new training methods (Blunsom and Kalchbrenner, 2013; Sutskever et al., 2014) have been proposed. The encoder-decoder approach for Neural Machine Translation (NMT) consists in encoding the source sentence into an intermediate vector repre- sentation and then generating (decoding) the target sentence from this representation. Cho et al. (2014) is an example of this approach. The NMT approach of jointly training alignment and translation models described by Bah- danau et al. (2014) and its successive variations have shown promising results. Its attention mechanism deals with the problem of having a fixed length vector for sentences of varying length by encoding the source sentence into a set of vectors, one vector for each of the tokens in the source sentence. NMT doesn’t need complex feature engineering, which is convenient when dealing with resource-limited languages. However, a large parallel corpus is still needed in order to get competitive performance and avoid overfitting. As an example, Bahdanau et al. (2014) and Jean et al. (2015) use a dataset of about 12 million parallel sentences. Two main problems arise when using a small dataset for training a MT model. One of those problems is that the vocabulary exposed by a small dataset is inherently small. Also, even if a word shows up in the data it may occur too few times for learning a reliable representation. One strategy for minimizing this problem is subdividing words into subword ������������������������������������������������������ ����������������������������������������� units, like syllables. Doing so reduces the total vocabulary size and increases the hit-rate of each symbol in the dataset. This has been explored in Sennrich et al. (2015b). Another problem concerns large NMT models. When the number of parameters is too large compared to the data size, the model may optimize for memorizing all or a large part …

Content truncated to 500 words