Integrating Translation Memories into Non-Autoregressive Machine Translation

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 1326–1338 May 2-6, 2023 ©2023 Association for Computational Linguistics Integrating Translation Memories into Non-Autoregressive Machine Translation Jitao Xu† Josep Crego‡ François Yvon† †Université Paris-Saclay, CNRS, LISN, 91400, Orsay, France ‡SYSTRAN, 5 rue Feydeau, 75002, Paris, France {jitao.xu,francois.yvon}@limsi.fr, josep.crego@systrangroup.com Abstract Non-autoregressive machine translation (NAT) has recently made great progress. However, most works to date have focused on standard translation tasks, even though some edit-based NAT models, such as the Levenshtein Trans- former (LevT), seem well suited to translate with a Translation Memory (TM). This is the scenario considered here. We first analyze the vanilla LevT model and explain why it does not do well in this setting. We then propose a new variant, TM-LevT, and show how to effec- tively train this model. By modifying the data presentation and introducing an extra deletion operation, we obtain performance that are on par with an autoregressive approach, while re- ducing the decoding load. We also show that incorporating TMs during training dispenses to use knowledge distillation, a well-known trick used to mitigate the multimodality issue. 1 Introduction Non-autoregressive neural machine translation (NAT) has been greatly advanced in recent years (Xiao et al., 2022). NAT takes advantage from par- allel decoding to generate multiple tokens simulta- neously and speed up inference. This is often at the cost of a loss in translation quality when compared to autoregressive (AR) models (Gu et al., 2018a). This gap is slowly closing and methods based on iterative refinement (Ghazvininejad et al., 2019; Gu et al., 2019; Saharia et al., 2020) and on con- nectionist temporal classification (Libovický and Helcl, 2018; Gu and Kong, 2021) are now reporting BLEU scores similar to strong AR baselines. Most works on NAT focus on the standard ma- chine translation (MT) task, where the decoder starts from scratch, with the exception of Susanto et al. (2020); Xu and Carpuat (2021), who use NAT to integrate lexical constraints in decoding. However, edit-based NAT models, such as the Lev- enshtein Transformer (LevT) of Gu et al. (2019), seem to be a natural candidate to perform MT with Translation Memories (TM). LevT is able to itera- tively edit an initial target sequence by performing insertion and deletion operations until convergence. This design also matches the concept of using TMs in MT, where given a source sentence, we aim to edit a candidate translation retrieved from the TM. This idea has been used for decades in the lo- calization industry and implemented into basic Computer-Aided Translation tools. Translators wishing to translate a sentence can benefit from fuzzy matching techniques to retrieve similar seg- ments from the TM. These segments can then be revised, thereby improving productivity and consis- tency of the translation process (Koehn and Senel- lart, 2010; Yamada, 2011). The retrieval of similar examples from a TM has also proved useful in con- ventional (AR) neural MT systems; they can be injected into the encoder (Bulte and Tezcan, 2019; Xu …

Content truncated to 500 words