Character-Aware Decoder for Translation into Morphologically Rich Languages

Character-Aware Decoder for Translation into Morphologically Rich Languages Adithya Renduchintala∗ and Pamela Shapiro∗ and Kevin Duh and Philipp Koehn Department of Computer Science Johns Hopkins University {adi.r,pshapiro,phi}@jhu.edu kevinduh@cs.jhu.edu Abstract Neural machine translation (NMT) sys- tems operate primarily on words (or sub- words), ignoring lower-level patterns of morphology. We present a character- aware decoder designed to capture such patterns when translating into morpho- logically rich languages. We achieve character-awareness by augmenting both the softmax and embedding layers of an attention-based encoder-decoder model with convolutional neural networks that operate on the spelling of a word. To in- vestigate performance on a wide variety of morphological phenomena, we translate English into 14 typologically diverse tar- get languages using the TED multi-target dataset. In this low-resource setting, the character-aware decoder provides consis- tent improvements with BLEU score gains of up to +3.05. In addition, we analyze the relationship between the gains obtained and properties of the target language and find evidence that our model does indeed exploit morphological patterns. 1 Introduction Traditional attention-based encoder-decoder neu- ral machine translation (NMT) models learn word- level embeddings, with a continuous representa- tion for each unique word type (Bahdanau et al., 2015). However, this results in a long tail of rare words for which we do not learn good representa- tions. More recently, it has become standard prac- c⃝ 2019 The authors. This article is licensed under a Creative Commons 4.0 licence, no derivative works, attribution, CC- BY-ND. ∗Equal Contribution tice to mitigate the vocabulary size problem with Byte-Pair Encoding (BPE) (Gage, 1994; Sennrich et al., 2016). BPE iteratively merges consecutive characters into larger chunks based on their fre- quency, which results in the breaking up of less common words into “subword units.” While BPE addresses the vocabulary size prob- lem, the spellings of the subword units are still ig- nored. On the other hand, purely character-level NMT translates one character at a time and can im- plicitly learn about morphological patterns within words as well as generalize to unseen vocabulary. Recently, Cherry et al. (2018) show that very deep character-level models can outperform BPE, how- ever, the smallest data size evaluated was 2 million sentences, so it is unclear if the results hold for low-resource settings and when translating into a range of different morphologically rich languages. Furthermore, tuning deep character-level models is expensive, even for low-resource settings.1 A middle-ground alternative is character-aware word-level modeling. Here, the NMT system op- erates over words but uses word embeddings that are sensitive to spellings and thereby has the abil- ity to learn morphological patterns in the language. Such character-aware approaches have been ap- plied successfully in NMT to the source-side word embedding layer (Costa-juss`a and Fonollosa, 2016), but surprisingly, similar gains have not been achieved on the target side (Belinkov et al., 2017). While source-side character-aware models only need to make the source embedding layer character-aware, on the target-side we require both the target embedding layer and the softmax layer 2 1The dropout rate was found to be critical …

Content truncated to 500 words