Linguistic Input Features Improve Neural Machine Translation

Proceedings of the First Conference on Machine Translation, Volume 1: Research Papers, pages 83–91, Berlin, Germany, August 11-12, 2016. c⃝2016 Association for Computational Linguistics Linguistic Input Features Improve Neural Machine Translation Rico Sennrich and Barry Haddow School of Informatics, University of Edinburgh rico.sennrich@ed.ac.uk, bhaddow@inf.ed.ac.uk Abstract Neural machine translation has recently achieved impressive results, while using little in the way of external linguistic in- formation. In this paper we show that the strong learning capability of neural MT models does not make linguistic fea- tures redundant; they can be easily incor- porated to provide further improvements in performance. We generalize the em- bedding layer of the encoder in the at- tentional encoder–decoder architecture to support the inclusion of arbitrary features, in addition to the baseline word feature. We add morphological features, part-of- speech tags, and syntactic dependency la- bels as input features to English↔German and English→Romanian neural machine translation systems. In experiments on WMT16 training and test sets, we find that linguistic input features improve model quality according to three metrics: per- plexity, BLEU and CHRF3. An open- source implementation of our neural MT system is available1, as are sample files and configurations2. 1 Introduction Neural machine translation has recently achieved impressive results (Bahdanau et al., 2015; Jean et al., 2015), while learning from raw, sentence- aligned parallel text and using little in the way of external linguistic information.3 However, we hypothesize that various levels of linguistic anno- tation can be valuable for neural machine trans- lation. Lemmatisation can reduce data sparse- 1https://github.com/rsennrich/nematus 2https://github.com/rsennrich/ wmt16-scripts 3Linguistic tools are most commonly used in preprocess- ing, e.g. for Turkish segmentation (Gülçehre et al., 2015). ness, and allow inflectional variants of the same word to explicitly share a representation in the model. Other types of annotation, such as parts- of-speech (POS) or syntactic dependency labels, can help in disambiguation. In this paper we in- vestigate whether linguistic information is benefi- cial to neural translation models, or whether their strong learning capability makes explicit linguistic features redundant. Let us motivate the use of linguistic features us- ing examples of actual translation errors by neu- ral MT systems. In translation out of English, one problem is that the same surface word form may be shared between several word types, due to homonymy or word formation processes such as conversion. For instance, close can be a verb, ad- jective, or noun, and these different meanings of- ten have distinct translations into other languages. Consider the following English→German exam- ple: 1. We thought a win like this might be close. 2. Wir dachten, dass ein solcher Sieg nah sein könnte. 3. *Wir dachten, ein Sieg wie dieser könnte schließen. For the English source sentence in Example 1 (our translation in Example 2), a neural MT sys- tem (our baseline system from Section 4) mis- translates close as a verb, and produces the Ger- man verb schließen (Example 3), even though close is an adjective in this sentence, which has the German translation nah. Intuitively, part- of-speech annotation of …

Content truncated to 500 words