Achieving Open Vocabulary Neural Machine Translation with Hybrid Word-Character Models

Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 1054–1063, Berlin, Germany, August 7-12, 2016. c⃝2016 Association for Computational Linguistics Achieving Open Vocabulary Neural Machine Translation with Hybrid Word-Character Models Minh-Thang Luong and Christopher D. Manning Computer Science Department, Stanford University, Stanford, CA 94305 {lmthang,manning}@stanford.edu Abstract Nearly all previous work on neural ma- chine translation (NMT) has used quite restricted vocabularies, perhaps with a subsequent method to patch in unknown words. This paper presents a novel word- character solution to achieving open vo- cabulary NMT. We build hybrid systems that translate mostly at the word level and consult the character components for rare words. Our character-level recur- rent neural networks compute source word representations and recover unknown tar- get words when needed. The twofold advantage of such a hybrid approach is that it is much faster and easier to train than character-based ones; at the same time, it never produces unknown words as in the case of word-based models. On the WMT’15 English to Czech translation task, this hybrid approach offers an ad- dition boost of +2.1−11.4 BLEU points over models that already handle unknown words. Our best system achieves a new state-of-the-art result with 20.7 BLEU score. We demonstrate that our character models can successfully learn to not only generate well-formed words for Czech, a highly-inflected language with a very complex vocabulary, but also build correct representations for English source words. 1 Introduction Neural Machine Translation (NMT) is a simple new architecture for getting machines to translate. At its core, NMT is a single deep neural network that is trained end-to-end with several advantages such as simplicity and generalization. Despite being relatively new, NMT has already achieved Figure 1: Hybrid NMT – example of a word- character model for translating “a cute cat” into “un joli chat”. Hybrid NMT translates at the word level. For rare tokens, the character-level compo- nents build source representations and recover tar- get <unk>. “_” marks sequence boundaries. state-of-the-art translation results for several lan- guage pairs such as English-French (Luong et al., 2015b), English-German (Jean et al., 2015a; Lu- ong et al., 2015a; Luong and Manning, 2015), and English-Czech (Jean et al., 2015b). While NMT offers many advantages over tra- ditional phrase-based approaches, such as small memory footprint and simple decoder implemen- tation, nearly all previous work in NMT has used quite restricted vocabularies, crudely treating all other words the same with an <unk> symbol. Sometimes, a post-processing step that patches in unknown words is introduced to alleviate this problem. Luong et al. (2015b) propose to annotate 1054 occurrences of target <unk> with positional infor- mation to track their alignments, after which sim- ple word dictionary lookup or identity copy can be performed to replace <unk> in the translation. Jean et al. (2015a) approach the problem similarly but obtain the alignments for unknown words from the attention mechanism. We refer to these as the unk replacement technique. Though simple, these approaches ignore several important properties of languages. First, monolin- gually, …

Content truncated to 500 words