Variable-Length Word Encodings for Neural Translation Models

Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2088–2093, Lisbon, Portugal, 17-21 September 2015. c⃝2015 Association for Computational Linguistics. Variable-Length Word Encodings for Neural Translation Models Rohan Chitnis and John DeNero Computer Science Division University of California, Berkeley {ronuchit,denero}@berkeley.edu Abstract Recent work in neural machine translation has shown promising performance, but the most effective architectures do not scale naturally to large vocabulary sizes. We propose and compare three variable-length encoding schemes that represent a large vocabulary corpus using a much smaller vocabulary with no loss in information. Common words are unaffected by our en- coding, but rare words are encoded us- ing a sequence of two pseudo-words. Our method is simple and effective: it requires no complete dictionaries, learning proce- dures, increased training time, changes to the model, or new parameters. Com- pared to a baseline that replaces all rare words with an unknown word symbol, our best variable-length encoding strategy im- proves WMT English-French translation performance by up to 1.7 BLEU. 1 Introduction Bahdanau et al. (2014) propose a neural transla- tion model that learns vector representations for in- dividual words as well as word sequences. Their approach jointly predicts a translation and a la- tent word-level alignment for a sequence of source words. However, the architecture of the network does not scale naturally to large vocabularies (Jean et al., 2014). In this paper, we propose a novel approach to circumvent the large-vocabulary challenge by pre- processing the source and target word sequences, encoding them as a longer token sequence drawn from a small vocabulary that does not discard any information. Common words are unaffected, but rare words are encoded as a sequence of two pseudo-words. The exact same learning and infer- ence machinery applied to these transformed data yields improved translations. We evaluate a family of 3 different encoding schemes based on Huffman codes. All of them eliminate the need to replace rare words with the unknown word symbol. Our approach is simpler than other methods recently proposed to address the same issue. It does not introduce new param- eters into the model, change the model structure, affect inference, require access to a complete dic- tionary, or require any additional learning proce- dures. Nonetheless, compared to a baseline system that replaces all rare words with an unknown word symbol, our encoding approach improves English- French news translation by up to 1.7 BLEU. 2 Background 2.1 Neural Machine Translation Neural machine translation describes approaches to machine translation that learn from corpora in a single integrated model that embeds words and sentences into a vector space (Kalchbrenner and Blunsom, 2013; Cho et al., 2014; Sutskever et al., 2014). We focus on one recent approach to neu- ral machine translation, proposed by Bahdanau et al. (2014), that predicts both a translation and its alignment to the source sentence, though our tech- nique is relevant to related approaches as well. The architecture consists of an encoder and a de- coder. The encoder receives a source sentence x …

Content truncated to 500 words