The Devil is in the Details: On the Pitfalls of Vocabulary Selection in Neural Machine Translation

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1861 - 1874 July 10-15, 2022 ©2022 Association for Computational Linguistics The Devil is in the Details: On the Pitfalls of Vocabulary Selection in Neural Machine Translation Tobias Domhan∗ Eva Hasler∗ Ke Tran Sony Trenous Bill Byrne Felix Hieber Amazon AI Translate Berlin, Germany {domhant,ehasler,trnke,trenous,willbyrn,fhieber}@amazon.com Abstract Vocabulary selection, or lexical shortlisting, is a well-known technique to improve latency of Neural Machine Translation models by con- straining the set of allowed output words dur- ing inference. The chosen set is typically determined by separately trained alignment model parameters, independent of the source- sentence context at inference time. While vo- cabulary selection appears competitive with respect to automatic quality metrics in prior work, we show that it can fail to select the right set of output words, particularly for se- mantically non-compositional linguistic phe- nomena such as idiomatic expressions, lead- ing to reduced translation quality as perceived by humans. Trading off latency for quality by increasing the size of the allowed set is often not an option in real-world scenarios. We propose a model of vocabulary selection, integrated into the neural translation model, that predicts the set of allowed output words from contextualized encoder representations. This restores translation quality of an uncon- strained system, as measured by human evalua- tions on WMT newstest2020 and idiomatic ex- pressions, at an inference latency competitive with alignment-based selection using aggres- sive thresholds, thereby removing the depen- dency on separately trained alignment models. 1 Introduction Neural Machine Translation (NMT) has achieved great improvements in translation quality, largely thanks to the introduction of Transformer mod- els (Vaswani et al., 2017). However, increasingly larger models (Aharoni et al., 2019; Arivazha- gan et al., 2019) lead to prohibitively slow infer- ence when deployed in industrial settings. Espe- cially for real-time applications, low latency is key. A number of inference optimization speed- ups have been proposed and are used in practice: ∗Equal contributions. EN: to swal DE: in den saulow the bitter pill ren ap fel bei ßen GL: to bite into the sour ap ple EN: by ho ok or cro DE: auf biegen und brechenok GL: by bending and breaking EN: to buy a p ig in a po ke DE: die kat ze im sack kaufen GL: to buy the cat in the bag EN: to swe at blood DE: blu t und wasser sch wit zen GL: to sweat blood and water EN: make yourself at home ! DE: machen sie es sich bequem ! GL: make yourself comfortable ! Figure 1: Examples of subword-segmented idiomatic expressions (EN) and their German correspondences (DE) as well as an English gloss (GL) of the German ex- pression. Alignment-based vocabulary selection: out- put tokens missing from the allowed set of top-k output tokens are marked in orange/bold (red/italic) for k=200 (k=1000). reduced precision (Aji and Heafield, 2020), replac- ing self-attention with Average Attention Networks (AANs) (Zhang et al., 2018), …

Content truncated to 500 words