Revisiting Locality Sensitive Hashing for Vocabulary Selection in Fast Neural Machine Translation

Proceedings of the Seventh Conference on Machine Translation (WMT), pages 855 - 869 December 7-8, 2022 ©2022 Association for Computational Linguistics Revisiting Locality Sensitive Hashing for Vocabulary Selection in Fast Neural Machine Translation Hieu Hoang∗ Marcin Junczys-Dowmunt∗ Roman Grundkiewicz Huda Khayrallah Microsoft, 1 Microsoft Way, Redmond, WA 98052, USA {hihoan,marcinjd,rogrundk,hkhayrallah}@microsoft.com Abstract Neural machine translation models often con- tain large target vocabularies. The calculation of logits, softmax and beam search is compu- tationally costly over so many classes. We investigate the use of locality sensitive hash- ing (LSH) to reduce the number of vocabulary items that must be evaluated and explore the re- lationship between the hashing algorithm, trans- lation speed and quality. Compared to prior work, our LSH-based solution does not require additional augmentation via word-frequency lists or alignments. We propose a training pro- cedure that produces models, which, when com- bined with our LSH inference algorithm in- crease translation speed by up to 87% over the baseline, while maintaining translation quality as measured by BLEU. Apart from just using BLEU, we focus on minimizing search errors compared to the full softmax, a much harsher quality criterion. 1 Introduction The computation of the output logit, softmax and beam search (the output layer) are some of the most compute-intensive tasks in current Neural Machine Translation (NMT) models, often taking the major- ity of inference time for many models, on many hardware architectures, especially in deployment settings. This is mainly due to the large vocabu- lary size relative to other dimensions in the model. Methods that reduce the effective vocabulary size can have a major impact on inference speed. Vo- cabulary selection is one such method. However, a known downside of vocabulary se- lection methods is the risk of search errors if the desired output token ˆyt is not a member of the re- duced vocabulary V , forcing the beam search to choose a sub-optimal token. Even when the impact of such search errors on BLEU is minimal, search errors caused by lexical shortlisting degrade human judgements of quality (Domhan et al., 2022). ∗These authors contributed equally to this work. Figure 1: Search errors vs. decrease in BLEU for all experiments in the paper. We vary the two main hy- perparameters of our LSH implementation: number of hash functions and the size k of our selected vocabulary subset. 60% of sentences have a search error before we observe a 1 BLEU point degradation. The degradation in human judgement is less sur- prising if we inspect Figure 1 which shows that translation quality, as measured by BLEU, is re- silient to search errors caused by LSH. Only when over 60% of translations exhibit search errors is there significant BLEU degradation. We examine vocabulary selection using Locality Sensitive Hashing (LSH), and evaluate specifically in the context of Neural Machine Translation. We introduce an LSH-based vocabulary selection algo- rithm and compatible models such that:1 1. the models have translation quality that is bet- ter than or comparable to the baseline model; 2. the LSH-based …

Content truncated to 500 words