Memory-augmented Neural Machine Translation

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1390–1399 Copenhagen, Denmark, September 7–11, 2017. c⃝2017 Association for Computational Linguistics Memory-augmented Neural Machine Translation Yang Feng2,1,4, Shiyue Zhang1,3, Andi Zhang1,3, Dong Wang1∗ and Andrew Abel5 1Center for Speech and Language Technologies, RIIT, Tsinghua University 2Key Laboratory of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences 3Beijing University of Posts and Telecommunications, China 4Huilan Corporation, Beijing, China 5Xi’An Jiaotong Liverpool-University, Suzhou, China fengyang@ict.ac.cn, {byryuer, andizhang912}@gmail.com wangdong99@mails.tsinghua.edu.cn, andrew.abel@xjtlu.edu.cn Abstract Neural machine translation (NMT) has achieved notable success in recent times, however it is also widely recognized that this approach has limitations with han- dling infrequent words and word pairs. This paper presents a novel memory- augmented NMT (M-NMT) architecture, which stores knowledge about how words (usually infrequently encountered ones) should be translated in a memory and then utilizes them to assist the neural model. We use this memory mechanism to com- bine the knowledge learned from a con- ventional statistical machine translation system and the rules learned by an NMT system, and also propose a solution for out-of-vocabulary (OOV) words based on this framework. Our experiments on two Chinese-English translation tasks demon- strated that the M-NMT architecture out- performed the NMT baseline by 9.0 and 2.7 BLEU points on the two tasks, respec- tively. Additionally, we found this archi- tecture resulted in a much more effective OOV treatment compared to competitive methods. 1 Introduction Neural Machine Translation (NMT) has been shown to have highly promising performance, par- ticularly when a large amount of training data is available (Wu et al., 2016; Johnson et al., 2016; Mi et al., 2016). Although there are different model architectures (Sutskever et al., 2014; Bahdanau et al., 2015), the common principle behind the NMT approach is the same: encoding the meaning of the input into a concept space and performing translation based on this encoding. This ‘meaning src. 人类共有二十三对染色体。 ref. Humans have 23 pairs of chromosomes. NMT There are 23-year history of human history. Table 1: An example of Chinese-to-English ‘meaning drift’ with NMT. encoding’ principle leads to a deeper understand- ing and learning of the translation rules, and hence a better translation than conventional statistic ma- chine translation (SMT) that considers only sur- face forms, i.e., words and phrases (Koehn et al., 2003). Despite positive results obtained so far, a par- ticular problem of the NMT approach is that it has a tendency towards overfitting to frequent ob- servations (words, word co-occurrences, transla- tion pairs, etc.), but overlooking special cases that are not frequently observed. For example, NMT is good at learning translation pairs that are fre- quently observed, and can make use of them well at run-time, but for low-frequency pairs in the training data, the system may ‘forget’ to use them when they should be. Unfortunately, rare words are inevitable for all translation tasks due to Zipf’s law, and indeed they are often the most impor- tant parts of a sentence, e.g., domain-specific en- tity names. Table …

Content truncated to 500 words