Neural Machine Translation with Monolingual Translation Memory

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 7307–7318 August 1–6, 2021. ©2021 Association for Computational Linguistics 7307 Neural Machine Translation with Monolingual Translation Memory∗ Deng Cai♥, Yan Wang♠, Huayang Li♠, Wai Lam♥, and Lemao Liu♠ ♥The Chinese University of Hong Kong thisisjcykcd@gmail.com wlam@se.cuhk.edu.hk ♠Tencent AI Lab {brandenwang,alanili,redmondliu}@tencent.com Abstract Prior work has proved that Translation mem- ory (TM) can boost the performance of Neural Machine Translation (NMT). In contrast to ex- isting work that uses bilingual corpus as TM and employs source-side similarity search for memory retrieval, we propose a new frame- work that uses monolingual memory and per- forms learnable memory retrieval in a cross- lingual manner. Our framework has unique ad- vantages. First, the cross-lingual memory re- triever allows abundant monolingual data to be TM. Second, the memory retriever and NMT model can be jointly optimized for the ulti- mate translation goal. Experiments show that the proposed method obtains substantial im- provements. Remarkably, it even outperforms strong TM-augmented NMT baselines using bilingual TM. Owning to the ability to lever- age monolingual data, our model also demon- strates effectiveness in low-resource and do- main adaptation scenarios. 1 Introduction Augmenting parametric neural network models with non-parametric memory (Khandelwal et al., 2019; Guu et al., 2020; Lewis et al., 2020a,b) has recently emerged as a promising direction to re- lieve the demand for ever-larger model size (De- vlin et al., 2019; Radford et al., 2019; Brown et al., 2020). For the task of Machine Translation (MT), inspired by the Computer-Aided Translation (CAT) tools by professional human translators for increas- ing productivity for decades (Yamada, 2011), the usefulness of Translation Memory (TM) has long been recognized (Huang et al., 2021). In general, TM is a database that stores pairs of source text and its corresponding translations. Like for human ∗The work described in this paper is partially supported by a grant from the Research Grant Council of the Hong Kong Special Administrative Region, China (Project Code: 14200719). Correspondence to Yan Wang. translation, early work (Koehn and Senellart, 2010; He et al., 2010; Utiyama et al., 2011; Wang et al., 2013, inter alia) presents translations for similar source input to statistical translation models as ad- ditional cues. Recent work has confirmed that TM can help Neural Machine Translation (NMT) models as well. In a similar spirit to prior work, TM-augmented NMT models do not discard the training corpus after training but keep exploiting it in the test time. These models perform translation in two stages: In the retrieval stage, a retriever searches for nearest neighbors (i.e., source-target pairs) from the train- ing corpus based on source-side similarity such as lexical overlaps (Gu et al., 2018; Zhang et al., 2018; Xia et al., 2019), embedding-based matches (Cao and Xiong, 2018), or a hybrid (Bulte and Tezcan, 2019; Xu et al., 2020); In the generation stage, the retrieved translations are injected into a standard NMT model by attending over them …

Content truncated to 500 words