Improving Retrieval Augmented Neural Machine Translation by Controlling Source and Fuzzy-Match Interactions

Findings of the Association for Computational Linguistics: EACL 2023, pages 289–295 May 2-6, 2023 ©2023 Association for Computational Linguistics Improving Retrieval Augmented Neural Machine Translation by Controlling Source and Fuzzy-Match Interactions Cuong Hoang∗, Devendra Sachan∗, Prashant Mathur, Brian Thompson, Marcello Federico AWS AI Labs pramathu@amazon.com Abstract We explore zero-shot adaptation, where a general-domain model has access to customer or domain specific parallel data at inference time, but not during training. We build on the idea of Retrieval Augmented Translation (RAT) where top-k in-domain fuzzy matches are found for the source sentence, and target- language translations of those fuzzy-matched sentences are provided to the translation model at inference time. We propose a novel archi- tecture to control interactions between a source sentence and the top-k fuzzy target-language matches, and compare it to architectures from prior work. We conduct experiments in two language pairs (En-De and En-Fr) by training models on WMT data and testing them with five and seven multi-domain datasets, respec- tively. Our approach consistently outperforms the alternative architectures, improving BLEU across language pair, domain, and number k of fuzzy matches with almost no trade-off on inference latency. 1 Introduction Domain adaptation techniques such as fine-tuning (Freitag and Al-Onaizan, 2016; Luong and Man- ning, 2015) are highly effective at increasing in- domain performance of neural machine translation (NMT) systems, but are impractical in many re- alistic settings. For example, consider a single machine serving translations to thousands of cus- tomers, each with a private Translation Memory (TM). In this case, adapting, storing and loading large adapted models for each customer is compu- tationally infeasible. In this paper we thus consider zero-shot adaptation instead, with a single general- domain model trained from heterogeneous sources that has access to the customer or domain specific TM only at inference time. ∗Work done while the authors were at AWS AI Labs. Our work builds on Retrieval Augmented Trans- lation (RAT) (Li et al., 2022; Bulte and Tezcan, 2019; Xu et al., 2020; He et al., 2021; Cai et al., 2021), a paradigm which combines a translation model (Vaswani et al., 2017) with an external re- triever module that retrieves the top-k most similar source sentences from a TM (i.e. "fuzzy matches") (Farajian et al., 2017; Gu et al., 2017; Bulte and Tezcan, 2019). The encoder encodes the input sen- tence along with the translations of the top-k fuzzy- matches and passes the resulting encodings to the decoder. Prior RAT methods for NMT have fallen into two camps: Early work (Bulte and Tezcan, 2019; Zhang et al., 2018) concatenated the source sen- tence and the top-k fuzzy matches before encoding, relying on the encoder’s self-attention to compare the source sentence to each target sentences and determine which target phrases are relevant for the translation. More recent work (He et al., 2021; Cai et al., 2021) has opted to encode the source sen- tences and the top-k fuzzy matches independently, effectively shifting the entire burden of determin- ing which target phrases are relevant to the decoder. …

Content truncated to 500 words