What Makes Word-level Neural Machine Translation Hard: A Case Study on English-German Translation

Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 3199–3208, Osaka, Japan, December 11-17 2016. What Makes Word-level Neural Machine Translation Hard: A Case Study on English-German Translation Fabian Hirschmann Jinseok Nam Johannes F¨urnkranz Knowledge Engineering Group Technische Universit¨at Darmstadt Darmstadt, Germany Abstract Traditional machine translation systems often require heavy feature engineering and the combi- nation of multiple techniques for solving different subproblems. In recent years, several end- to-end learning architectures based on recurrent neural networks have been proposed. Unlike traditional systems, Neural Machine Translation (NMT) systems learn the parameters of the model and require only minimal preprocessing. Memory and time constraints allow to take only a fixed number of words into account, which leads to the out-of-vocabulary (OOV) prob- lem. In this work, we analyze why the OOV problem arises and why it is considered a serious problem in German. We study the effectiveness of compound word splitters for alleviating the OOV problem, resulting in a 2.5+ BLEU points improvement over a baseline on the WMT’14 German-to-English translation task. For English-to-German translation, we use target-side com- pound splitting through a special syntax during training that allows the model to merge compound words and gain 0.2 BLEU points. 1 Introduction In the field of machine translation, traditional methods depend on a careful design of useful features obtained by analyzing linguistic properties of the translation tasks. Inspired by recent success of meth- ods that learn vector representation of words from large corpora in an unsupervised way, Devlin et al. (2014) have shown that word representations learned by neural language models can be effectively used as components in a complex translation system. By contrast, Neural Machine Translation (NMT) models are end-to-end learning systems that tune model parameters so that the desired output sentence is gen- erated for a given input sentence of arbitrary length with minimal processing steps such as tokenization. Most of recently proposed NMT models are based on the encoder-decoder framework (Sutskever et al., 2014; Cho et al., 2014), where a source sentence is mapped into a fixed-size vector that preserves both the syntactic and semantic structure of the source sentence, from which a decoder starts with generating a sequence of words in a target language. Bahdanau et al. (2015) extend the vanilla encoder-decoder NMT framework by adding a small feed-forward neural network which learns which word in the source sentence is relevant for predicting the next word in the target sequence. It has been shown that the performance of such soft-attention NMTs stays consistent as the sequence length increases. Although NMT models have been applied successfully to translation tasks, it is still challenging to handle out-of-vocabulary (OOV) words because only a small number of words are considered to reduce computational overhead. All words not in the vocabulary are assigned to a single special token, e.g., ⟨UNK⟩ in our case. In order to address the OOV problem, Jean et al. (2015) further extend the model of Bahdanau et al. (2015) with importance sampling so that …

Content truncated to 500 words