Fast Nearest Neighbor Machine Translation

Findings of the Association for Computational Linguistics: ACL 2022, pages 555 - 565 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Fast Nearest Neighbor Machine Translation Yuxian Meng♣, Xiaoya Li♣, Xiayu Zheng▲, Fei Wu♠n Xiaofei Sun♠♣, Tianwei Zhang♦, Jiwei Li♠♣ ♠College of Computer Science and Technology, Zhejiang University, ♣Shannon.AI ▲Peking University, ♦Nanyang Technological University nShanghai Institute for Advanced Study of Zhejiang University, nShanghai AI Laboratory {yuxian_meng, xiaoya_li, xiaofei_sun, jiwei_li}@shannonai.com xiayu_zheng@pku.edu.cn, tianwei.zhang@ntu.edu.sg, wufei@zju.edu.cn Abstract Though nearest neighbor Machine Transla- tion (kNN-MT) (Khandelwal et al., 2020) has proved to introduce significant performance boosts over standard neural MT systems, it is prohibitively slow since it uses the entire ref- erence corpus as the datastore for the nearest neighbor search. This means each step for each beam in the beam search has to search over the entire reference corpus. kNN-MT is thus two-orders slower than vanilla MT mod- els, making it hard to be applied to real-world applications, especially online services. In this work, we propose Fast kNN-MT to address this issue. Fast kNN-MT constructs a signif- icantly smaller datastore for the nearest neigh- bor search: for each word in a source sentence, Fast kNN-MT first selects its nearest token- level neighbors, which is limited to tokens that are the same as the query token. Then at each decoding step, in contrast to using the entire corpus as the datastore, the search space is limited to target tokens corresponding to the previously selected reference source tokens. This strategy avoids search through the whole datastore for nearest neighbors and drastically improves decoding efficiency. Without loss of performance, Fast kNN-MT is two-orders faster than kNN-MT, and is only two times slower than the standard NMT model. Fast kNN-MT enables the practical use of kNN- MT systems in real-world MT applications. 1 1 Introduction Machine translation (MT) is a fundamental task in natural language processing (Brown et al., 1993; Och and Ney, 2003), and the prevalence of deep neural networks has spurred a diverse array of neu- ral machine translation (NMT) models to improve translation quality (Sutskever et al., 2014; Bah- danau et al., 2014; Vaswani et al., 2017). The recently proposed k nearest neighbor (kNN) MT 1The code is available at https://github.com/ ShannonAI/fast-knn-nmt model (Khandelwal et al., 2020) has proved to in- troduce significant performance boosts over stan- dard neural MT systems. The basic idea behind kNN-MT is that at each decoding step, the model is allowed to refer to reference target tokens with similar translation contexts in a large datastore of cached examples. The corresponding reference tar- get tokens provide important insights on the trans- lation token likely to appear next. One notable limitation of kNN-MT is that it is prohibitively slow: it uses the entire reference corpus as the datastore for the nearest neighbor search. This means each step for each beam in the beam search has to search over the entire reference corpus. kNN-MT is thus two-orders slower than vanilla MT models. The original paper of kNN- MT (Khandelwal et al., 2020) suggests …

Content truncated to 500 words