Empirical Assessment of kNN-MT for Real-World Translation Scenarios

Empirical Assessment of kNN-MT for Real-World Translation Scenarios Pedro Henrique Martins∗1, Jo˜ao Alves∗1, Tˆania Vaz1, Madalena Gonc¸alves1, Beatriz Silva1, Marianna Buchicchio1, Jos´e G. C. de Souza1, Andr´e F. T. Martins1,2,3 1Unbabel, Lisbon, Portugal, 2Instituto de Telecomunicac¸˜oes, Lisbon, Portugal 3Instituto Superior T´ecnico, University of Lisbon, Portugal Abstract This paper aims to investigate the effec- tiveness of the k-Nearest Neighbor Ma- chine Translation model (kNN-MT) in real-world scenarios. kNN-MT is a retrieval-augmented framework that com- bines the advantages of parametric mod- els with non-parametric datastores built us- ing a set of parallel sentences. Previous studies have primarily focused on evaluat- ing the model using only the BLEU met- ric and have not tested kNN-MT in real- world scenarios. Our study aims to fill this gap by conducting a comprehensive analy- sis on various datasets comprising different language pairs and different domains, us- ing multiple automatic metrics and expert- evaluated Multidimensional Quality Met- rics (MQM). We compare kNN-MT with two alternate strategies: fine-tuning all the model parameters and adapter-based fine- tuning. Finally, we analyze the effect of the datastore size on translation quality, and we examine the number of entries neces- sary to bootstrap and configure the index. 1 Introduction The remarkable advances in neural models have brought significant progress in the field of machine translation (Sutskever et al., 2014; Bahdanau et al., 2015; Vaswani et al., 2017). However, current sys- tems rely heavily on a fully-parametric approach, where the entire training data is compressed into ∗Equal contribution. Contact: {pedro.martins, joao.alves}@unbabel.com. ∗© 2023 The authors. This article is licensed under a Creative Commons 4.0 licence, no derivative works, attribution, CC- BY-ND. the model parameters. This can lead to inade- quate translations when encountering rare words or sentences outside of the initial training do- main (Koehn and Knowles, 2017), requiring sev- eral stages of fine-tuning to adapt to data drift or to new domains. By combining the advantages of parametric models with non-parametric databases built from parallel sentences, retrieval-augmented models showed to be a promising solution, particularly in domain adaptation scenarios (Gu et al., 2018; Zhang et al., 2018; Bapna and Firat, 2019; Meng et al., 2021; Zheng et al., 2021; Jiang et al., 2021; Martins et al., 2022a; Martins et al., 2022b). One notable example is the k-Nearest Neighbor Machine Translation model (kNN-MT) (Khandel- wal et al., 2021), known for its simplicity and very promising results. The model first creates a token- level datastore using parallel sentences, and then it retrieves similar examples from the database dur- ing inference, enhancing the generation process via interpolation of probability distributions. However, despite its potential, the kNN-MT model has yet to be tested in real-world scenar- ios. Previous studies have primarily focused on evaluating it using only the BLEU metric, which correlates poorly with human judgments. In or- der to gain a deeper understanding of when and how kNN-MT can be effective, we conduct a thorough analysis on various datasets which com- prise 4 different language pairs and 3 different do- mains, using BLEU (Papineni et …

Content truncated to 500 words