Learning Decoupled Retrieval Representation for Nearest Neighbour Neural Machine Translation

Proceedings of the 29th International Conference on Computational Linguistics, pages 5142–5147 October 12–17, 2022. 5142 Learning Decoupled Retrieval Representation for Nearest Neighbour Neural Machine Translation Qiang Wang1,2, Rongxiang Weng3,4, Ming Chen2∗ 1Zhejiang University, Hangzhou, China 2RoyalFlush AI Research Insistute, Hangzhou, China 3School of Computer Science and Technology, Soochow University, Suzhou, China 4miHoYo AI, Shanghai, China {wangqiangenu, wengrongxiang}@gmail.com, chenming@myhexin.com Abstract K-Nearest Neighbor Neural Machine Transla- tion (kNN-MT) successfully incorporates exter- nal corpus by retrieving word-level representa- tions at test time. Generally, kNN-MT borrows the off-the-shelf context representation in the translation task, e.g., the output of the last de- coder layer, as the query vector of the retrieval task. In this work, we highlight that coupling the representations of these two tasks is sub- optimal for fine-grained retrieval. To alleviate it, we leverage supervised contrastive learning to learn the distinctive retrieval representation derived from the original context representation. We also propose a fast and effective approach to constructing hard negative samples. Exper- imental results on five domains show that our approach improves the retrieval accuracy and BLEU score compared to vanilla kNN-MT. 1 Introduction Conventional neural machine translation (NMT) cannot dynamically incorporate external corpus at inference once finishing training (Bahdanau et al., 2015; Vaswani et al., 2017), resulting in bad perfor- mance when facing unseen domains, even if feed- ing millions or billions of sentence pairs for train- ing (Koehn and Knowles, 2017). To address this problem, researchers developed retrieval-enhanced NMT (RENMT) to flexibly incorporate external translation knowledge. Early RENMTs leverage a search engine to find the similar bitext to improve the translation performance (Zhang et al., 2018; Cao and Xiong, 2018; Gu et al., 2018; Xia et al., 2019). However, the results of sentence-level re- trieval with high similarity are generally sparse in practical applications, while noises in low simi- larity retrieval could lead to severe performance degradation (Cao and Xiong, 2018). kNN-MT proposed by Khandelwal et al. (2021) effectively alleviates the sparse problem by intro- ∗Corresponding author. ducing the word-level k-nearest neighbor mecha- nism. Instead of storing the discrete word sequence, kNN-MT uses a pre-trained NMT model to force decoding the external corpus and remembers the word-level continuous context representation, e.g., the output of the last decoder layer. During infer- ence, kNN-MT assumes that the same target words have similar contextual representations and weights word selection through retrieving current context representation from the memorized datastore. How- ever, we point out that it is sub-optimal to directly use the off-the-shelf context representation in the translation task because this vector is not specific to fine-grained retrieval. In this work, we attempt to decouple the context representation by learning an independent retrieval representation. To this end, we leverage supervised contrastive learning with multiple positive and neg- ative samples to learn a good retrieval representa- tion (called CLKNN). We also propose a fast and effective method to construct hard negative sam- ples. Experimental results on five domains show that our approach outperforms the vanilla kNN-MT in terms of BLEU and retrieval accuracy. 2 Background …

Content truncated to 500 words