Look It Up: Bilingual and Monolingual Dictionaries Improve Neural Machine Translation

This paper was retracted. For more information, see https://aclanthology.org/2020.wmt-1.65. Proceedings of the 5th Conference on Machine Translation (WMT), pages 538–549 Online, November 19–20, 2020. c⃝2020 Association for Computational Linguistics 538 Look It Up: Bilingual and Monolingual Dictionaries Improve Neural Machine Translation Xing Jie Zhong University of Notre Dame xzhong3@nd.edu David Chiang University of Notre Dame dchiang@nd.edu Abstract Despite advances in neural machine transla- tion (NMT) quality, rare words continue to be problematic. For humans, the solution to the rare-word problem has long been dictionaries, but dictionaries cannot be straightforwardly in- corporated into NMT. In this paper, we de- scribe a new method for “attaching” dictionary definitions to rare words so that the network can learn the best way to use them. We demon- strate improvements of up to 3.1 BLEU using bilingual dictionaries and up to 0.7 BLEU us- ing monolingual source-language dictionaries. 1 Introduction Despite its successes, neural machine translation (NMT) still has unresolved problems. Among them is the problem of rare words, which are paradoxi- cally very common because of Zipf’s Law. In part, this is a problem intrinsic to data-driven machine translation because the system will inevitably en- counter words not seen in the training data. In part, however, NMT systems seem particularly chal- lenged by rare words, compared with older sta- tistical models. One reason is that NMT systems have a fixed- size vocabulary, typically 10k–100k words; words outside this vocabulary are represented using a spe- cial symbol like UNK. Byte pair encoding (BPE) breaks rare words into smaller, more frequent sub- words, at least allowing NMT to see them instead of UNK (Sennrich et al., 2016). But this by no means solves the problem; even with subwords, NMT seems to have difficulty learning translations of very rare words, possibly an instance of catas- trophic forgetting (McCloskey and Cohen, 1989). Humans deal with rare words by looking them up in a dictionary, and the idea of using dictionaries to assist machine translation is extremely old. From a statistical perspective, dictionaries are a useful complement to running text because the uniform distribution of dictionary headwords can smooth out the long-tailed distribution of running text. In pre-neural statistical machine translation systems, the typical way to incorporate bilingual dictionaries is simply to include them as parallel sentences in the training data. But (as we show), this does not work well for NMT systems. We are aware of only a few previous attempts to find better ways to incorporate bilingual dic- tionaries in NMT. Some methods use dictionaries to synthesize new training examples (Zhang and Zong, 2016; Qi et al., 2018; H¨am¨al¨ainen and Alna- jjar, 2019). Arthur et al. (2016) extend the model to encourage it to generate translations from the (automatically extracted) dictionary. Post and Vilar (2018) constrain the decoder to generate transla- tions from the dictionary. What these approaches have in common is that they all treat dictionary def- initions as target-language text, when, in fact, they often have properties very different from ordinary text. For example, CEDICT …

Content truncated to 500 words