Neural Machine Translation of Rare Words with Subword Units

Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 1715–1725, Berlin, Germany, August 7-12, 2016. c⃝2016 Association for Computational Linguistics Neural Machine Translation of Rare Words with Subword Units Rico Sennrich and Barry Haddow and Alexandra Birch School of Informatics, University of Edinburgh {rico.sennrich,a.birch}@ed.ac.uk, bhaddow@inf.ed.ac.uk Abstract Neural machine translation (NMT) mod- els typically operate with a fixed vocabu- lary, but translation is an open-vocabulary problem. Previous work addresses the translation of out-of-vocabulary words by backing off to a dictionary. In this pa- per, we introduce a simpler and more ef- fective approach, making the NMT model capable of open-vocabulary translation by encoding rare and unknown words as se- quences of subword units. This is based on the intuition that various word classes are translatable via smaller units than words, for instance names (via character copying or transliteration), compounds (via com- positional translation), and cognates and loanwords (via phonological and morpho- logical transformations). We discuss the suitability of different word segmentation techniques, including simple character n- gram models and a segmentation based on the byte pair encoding compression algo- rithm, and empirically show that subword models improve over a back-off dictionary baseline for the WMT 15 translation tasks English→German and English→Russian by up to 1.1 and 1.3 BLEU, respectively. 1 Introduction Neural machine translation has recently shown impressive results (Kalchbrenner and Blunsom, 2013; Sutskever et al., 2014; Bahdanau et al., 2015). However, the translation of rare words is an open problem. The vocabulary of neu- ral models is typically limited to 30 000–50 000 words, but translation is an open-vocabulary prob- The research presented in this publication was conducted in cooperation with Samsung Electronics Polska sp. z o.o. - Samsung R&D Institute Poland. lem, and especially for languages with produc- tive word formation processes such as aggluti- nation and compounding, translation models re- quire mechanisms that go below the word level. As an example, consider compounds such as the German Abwasser|behandlungs|anlange ‘sewage water treatment plant’, for which a segmented, variable-length representation is intuitively more appealing than encoding the word as a fixed-length vector. For word-level NMT models, the translation of out-of-vocabulary words has been addressed through a back-off to a dictionary look-up (Jean et al., 2015; Luong et al., 2015b). We note that such techniques make assumptions that often do not hold true in practice. For instance, there is not al- ways a 1-to-1 correspondence between source and target words because of variance in the degree of morphological synthesis between languages, like in our introductory compounding example. Also, word-level models are unable to translate or gen- erate unseen words. Copying unknown words into the target text, as done by (Jean et al., 2015; Luong et al., 2015b), is a reasonable strategy for names, but morphological changes and transliteration is often required, especially if alphabets differ. We investigate NMT models that operate on the level of subword units. Our main goal is to model open-vocabulary translation in the NMT network itself, without requiring a back-off model for rare …

Content truncated to 500 words