Efficient Inference for Multilingual Neural Machine Translation

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 8563–8583 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 8563 Efficient Inference for Multilingual Neural Machine Translation Alexandre Bérard∗ NAVER LABS Europe Dain Lee† NAVER Corp. Stéphane Clinchant⋆ NAVER LABS Europe Kweonwoo Jung‡ NAVER Corp. Vassilina Nikoulina⋆ NAVER LABS Europe Abstract Multilingual NMT has become an attractive so- lution for MT deployment in production. But to match bilingual quality, it comes at the cost of larger and slower models. In this work, we consider several ways to make mul- tilingual NMT faster at inference without de- grading its quality. We experiment with sev- eral “light decoder" architectures in two 20- language multi-parallel settings: small-scale on TED Talks and large-scale on ParaCrawl. Our experiments demonstrate that combining a shallow decoder with vocabulary filtering leads to more than ×2 faster inference with no loss in translation quality. We validate our find- ings with BLEU and chrF (on 380 language pairs), robustness evaluation and human evalu- ation. 1 Introduction Multilingual machine translation (Johnson et al., 2017; Bapna and Firat, 2019; Aharoni et al., 2019; Zhang et al., 2020; Fan et al., 2020; Lyu et al., 2020) has made a lot of progress in the last years. It is attractive because it allows handling multiple language directions within a single model, thus significantly reducing training and maintenance costs. However, to preserve good performance across all the language pairs, both the vocabulary size and model size have to be increased compared to bilingual NMT, which hurts inference speed. For example, the recently released M2M-100 (Fan et al., 2020) has 15B parameters and needs multi- ple GPUs for inference. The problem of inference speed has been well studied in bilingual settings (Kasai et al., 2021a,b; Chen et al., 2018; Hsu et al., 2020; Kim et al., 2019; Li et al., 2020; Shi and Knight, 2017). One line of work consists in us- ing lighter decoder architectures (e.g., shallow de- coder: Kasai et al., 2021a, RNN decoder: Chen ∗first.last@naverlabs.com †dain.l@navercorp.com ‡kweonwoo.jung@navercorp.com et al., 2018; Kim et al., 2019; Lei, 2021). These works demonstrate that it is possible to significantly speed up inference with almost no loss in transla- tion quality as measured by BLEU. Decoding time (sec) biling. multi. 0 25 50 75 100 dec w/o softmax enc total other beam top k softmax Figure 1: Decoding time spent in different components of bilingual (vocab size 16k) vs multilingual (vocab size 69k) Transformer Big 6-6 models on TED2020 valid EN-DE (average over 10 runs). Figure 1 compares the inference time spent in each NMT component by bilingual and multilin- gual models of the same architecture but with dif- ferent vocabulary sizes. The decoder is also the bottleneck in the multilingual model which sug- gests that we can expect similar speed gains with a lighter decoder. It also indicates that some speed gain could be obtained by reducing vocabulary size (which impacts both beam search and softmax). However, it is not so obvious that lighter …

Content truncated to 500 words