Bridging the Domain Gaps in Context Representations for k-Nearest Neighbor Neural Machine Translation

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 5841–5853 July 9-14, 2023 ©2023 Association for Computational Linguistics Bridging the Domain Gaps in Context Representations for k-Nearest Neighbor Neural Machine Translation Zhiwei Cao1,3∗, Baosong Yang2, Huan Lin2, Suhang Wu1, Xiangpeng Wei2 Dayiheng Liu2, Jun Xie2, Min Zhang4 and Jinsong Su1,3† 1School of Informatics, Xiamen University, China 2Language Technology Lab, Alibaba DAMO Academy 3Institute of Artificial Intelligence, Xiamen University, China 4Institute of Computer Science and Technology, Soochow University, China lines1@stu.xmu.edu.cn, jssu@xmu.edu.cn Abstract k-Nearest neighbor machine translation (kNN- MT) has attracted increasing attention due to its ability to non-parametrically adapt to new trans- lation domains. By using an upstream NMT model to traverse the downstream training cor- pus, it is equipped with a datastore contain- ing vectorized key-value pairs, which are re- trieved during inference to benefit translation. However, there often exists a significant gap between upstream and downstream domains, which hurts the retrieval accuracy and the final translation quality. To deal with this issue, we propose a novel approach to boost the datastore retrieval of kNN-MT by reconstructing the orig- inal datastore. Concretely, we design a reviser to revise the key representations, making them better fit for the downstream domain. The re- viser is trained using the collected semantically- related key-queries pairs, and optimized by two proposed losses: one is the key-queries seman- tic distance ensuring each revised key represen- tation is semantically related to its correspond- ing queries, and the other is an L2-norm loss encouraging revised key representations to ef- fectively retain the knowledge learned by the upstream NMT model. Extensive experiments on domain adaptation tasks demonstrate that our method can effectively boost the datastore retrieval and translation quality of kNN-MT.1 1 Introduction The recently proposed k-Nearest Neighbors Ma- chine Translation (kNN-MT) (Khandelwal et al., 2021) is increasingly receiving attention from the community of machine translation due to its advan- tage on non-parametric domain adaptation (Zheng et al., 2021a; Wang et al., 2022a; Meng et al., 2022). Given an upstream NMT model, kNN-MT first uses ∗ This work was done when Zhiwei Cao was interning at DAMO Academy, Alibaba Group. † Corresponding author. 1Our code is available at https://github.com/ DeepLearnXMU/RevisedKey-knn-mt. Figure 1: An example of datastore retrieval, where News and IT are the upstream and downstream domains, re- spectively. We first build a downstream NMT model by fine-tuning the upstream NMT model on the down- stream training corpus. Then, we use the downstream NMT model to re-traverse the downstream training cor- pus, constructing a downstream datastore. Finally, we reuse the upstream and downstream NMT model to con- duct retrieval on the original and downstream datastores, respectively. The result shows that the nearest neighbors retrieved by the same query are quite different, and only the retrieved nearest neighbors from the downstream datastore contain the ground-truth token “spreadsheet”. the downstream training corpus to establish a data- store containing key-value pairs, where each key is the representation of the NMT decoder and its value …

Content truncated to 500 words