Towards Robust k-Nearest-Neighbor Machine Translation

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5468 - 5477 December 7-11, 2022 ©2022 Association for Computational Linguistics Towards Robust k-Nearest-Neighbor Machine Translation Hui Jiang1∗, Ziyao Lu2∗, Fandong Meng2, Chulun Zhou2, Jie Zhou2, Degen Huang3 and Jinsong Su1,4† 1School of Informatics, Xiamen University, China 2Pattern Recognition Center, WeChat AI, Tencent Inc, China 3Dalian University of Technology, China 4Pengcheng Laboratory, China hjiang@stu.xmu.edu.cn {ziyaolu,fandongmeng,chulunzhou,withtomzhou}@tencent.com huangdg@dlut.edu.cn jssu@xmu.edu.cn Abstract k-Nearest-Neighbor Machine Translation (kNN-MT) becomes an important research direction of NMT in recent years. Its main idea is to retrieve useful key-value pairs from an additional datastore to modify translations without updating the NMT model. However, the underlying retrieved noisy pairs will dra- matically deteriorate the model performance. In this paper, we conduct a preliminary study and find that this problem results from not fully exploiting the prediction of the NMT model. To alleviate the impact of noise, we propose a confidence-enhanced kNN-MT model with robust training. Concretely, we introduce the NMT confidence to refine the modeling of two important components of kNN-MT: kNN distribution and the interpolation weight. Meanwhile we inject two types of perturbations into the retrieved pairs for robust training. Experimental results on four benchmark datasets demonstrate that our model not only achieves significant improvements over current kNN-MT models, but also exhibits better robustness. Our code is available at https://github.com/ DeepLearnXMU/Robust-knn-mt. 1 Introduction As a commonly-used paradigm of retrieval-based neural machine translation (NMT), k-Nearest- Neighbor Machine Translation (kNN-MT) has proven to be effective in many studies (Khandel- wal et al., 2021; Zheng et al., 2021a; Jiang et al., 2021; Wang et al., 2022; Meng et al., 2022), and thus attracted much attention in the community of machine translation. The core of kNN-MT is to use an auxiliary datastore containing cached decoder representations and corresponding target tokens. This datastore can flexibly guide the NMT model This work is done when Hui Jiang was interning at Pattern Recognition Center, WeChat AI, Tencent Inc, China. *Equal contribution †Corresponding author Distance Value Key Final Distribution Estimate I have been … … no…been… 2 209 33 no been a my … you summer Decoder Output: NMT Model … no…been… 𝑝��� 𝑝��� … no…been… 𝜆 = 0.7 J'ai été dans ma propre chambre. … you bee a my you … summer … Datastore kNN Distribution NMT Distribution Source: Decoder Representation (1 − 𝜆) 𝜆 Value Value Retrieved Pairs Key ‘‘no’’ Figure 1: An example of kNN-MT model using dynam- ically estimation of λ (Zheng et al., 2021a; Jiang et al., 2021). to make better predictions, especially for domain adaptation. Compared with other retrieval-based paradigm (Tu et al., 2018; Cai et al., 2021), kNN- MT has two advantages: 1) It is more scalable because we can directly improve the NMT model by just manipulating the datastore. 2) It is more interpretable due to its observable retrieved pairs. Generally, kNN-MT mainly involves two stages: datastore establishment and candidate retrieval. During the first stage, a pre-trained NMT model is used to construct a datastore containing …

Content truncated to 500 words