Non-Autoregressive Machine Translation: It’s Not as Fast as it Seems

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1780 - 1790 July 10-15, 2022 ©2022 Association for Computational Linguistics Non-Autoregressive Machine Translation: It’s Not as Fast as it Seems Jindˇrich Helcl1,2 and Barry Haddow1 and Alexandra Birch1 1School of Informatics, University of Edinburgh 2Faculty of Mathematics and Physics, Charles University {jhelcl,bhaddow,a.birch}@ed.ac.uk Abstract Efficient machine translation models are com- mercially important as they can increase infer- ence speeds, and reduce costs and carbon emis- sions. Recently, there has been much interest in non-autoregressive (NAR) models, which promise faster translation. In parallel to the research on NAR models, there have been suc- cessful attempts to create optimized autore- gressive models as part of the WMT shared task on efficient translation. In this paper, we point out flaws in the evaluation methodology present in the literature on NAR models and we provide a fair comparison between a state- of-the-art NAR model and the autoregressive submissions to the shared task. We make the case for consistent evaluation of NAR models, and also for the importance of comparing NAR models with other widely used methods for im- proving efficiency. We run experiments with a connectionist-temporal-classification-based (CTC) NAR model implemented in C++ and compare it with AR models using wall clock times. Our results show that, although NAR models are faster on GPUs, with small batch sizes, they are almost always slower under more realistic usage conditions. We call for more realistic and extensive evaluation of NAR models in future work. 1 Introduction Non-autoregressive neural machine translation (NAR NMT, or NAT; Gu et al., 2018; Lee et al., 2018) is an emerging subfield of NMT which fo- cuses on increasing the translation speed by chang- ing the model architecture. The defining feature of non-autoregressive mod- els is the conditional independence assumption on the output probability distributions; this is in con- trast to autoregressive models, where the output distributions are conditioned on the previous out- puts. This conditional independence allows one to decode the target tokens in parallel. This can substantially reduce the decoding time, especially for longer target sentences. The decoding speed is assessed by translating a test set and measuring the overall time the process takes. This may sound simple, but there are various aspects to be considered that can affect decoding speed, such as batching, number of hypotheses in beam search or hardware used (i.e., using CPU or GPU). Decoding speed evaluation is a challeng- ing task, especially when it comes to comparabil- ity across different approaches. Unlike translation quality, decoding speed can be measured exactly. However, also unlike translation quality, different results are obtained from the same system under different evaluation environments. The WMT Ef- ficient Translation Shared Task aims to evaluate efficiency research and encourages the reporting of a range of speed and translation quality values to better understand the trade-off across different model configurations (Heafield et al., 2021). In this paper, we follow the emerging best practices de- veloped …

Content truncated to 500 words