Word Representations in Factored Neural Machine Translation

Proceedings of the Conference on Machine Translation (WMT), Volume 1: Research Papers, pages 20–31 Copenhagen, Denmark, September 711, 2017. c⃝2017 Association for Computational Linguistics Word Representations in Factored Neural Machine Translation Franck Burlot∗ LIMSI, CNRS, Universit´e Paris Saclay Mercedes Garc´ıa-Mart´ınez∗ LIUM, University of Le Mans Lo¨ıc Barrault LIUM, University of Le Mans Fethi Bougares LIUM, University of Le Mans Franc¸ois Yvon LIMSI, CNRS, Universit´e Paris Saclay Abstract Translation into a morphologically rich language requires a large output vocabu- lary to model various morphological phe- nomena, which is a challenge for neu- ral machine translation architectures. To address this issue, the present paper in- vestigates the impact of having two out- put factors with a system able to generate separately two distinct representations of the target words. Within this framework, we investigate several word representa- tions that correspond to different distri- butions of morpho-syntactic information across both factors. We report experiments for translation from English into two mor- phologically rich languages, Czech and Latvian, and show the importance of ex- plicitly modeling target morphology. 1 Introduction Open vocabularies remain a challenge for Neu- ral Machine Translation (NMT) (Cho et al., 2014; Bahdanau et al., 2015), both for linguistic and computational reasons. From a linguistic stand- point, morphological variation and lexical produc- tivity cause word forms unseen in training to oc- cur in source texts, which may also require to gen- erate novel target word forms. Using very large input/output vocabularies partially mitigates these issues, yet may cause serious instability (when computing embeddings of rare or unseen words) and complexity issues (when dealing with large softmax layers). Several proposals have been put forward to ad- dress these problems, which are particularly harm- ful when one language is a morphologically rich ∗Both authors have contributed equally to this work. language (MRL), exhibiting larger token/type ra- tio than is observed for English. One strategy is to improve NMT’s internal procedures: for instance by using a structured output layer (Mnih and Hin- ton, 2008) or by altering the training or decoding criteria (Jean et al., 2015). An alternative approach is to work with representations designed to remove some variations via source-side or target-side nor- malization procedures; or more radically to con- sider character-based representations (Ling et al., 2015; Luong and Manning, 2016; Costa-juss`a and Fonollosa, 2016), which are however much more costly to train, and make long distance dependen- cies even longer. None has however been as successful as the re- cent proposal of Sennrich et al. (2016b) which seems to achieve a right balance between a lim- ited vocabulary size and an ability to translate a fully open vocabulary. In a nutshell, this approach decomposes source and target tokens into smaller units of variable length (using what is now termed as a “Byte Pair Encoding” or BPE in short): this means that (a) all source tokens can be represented as a sequence of such units, which crucially are all seen in training; (b) all possible target words can also be generated; (c) the size …

Content truncated to 500 words