Accelerating Transformer Inference for Translation via Parallel Decoding

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 12336–12355 July 9-14, 2023 ©2023 Association for Computational Linguistics Accelerating Transformer Inference for Translation via Parallel Decoding Andrea Santilli1, Silvio Severino1, Emilian Postolache1, Valentino Maiorca1, Michele Mancusi1, Riccardo Marin2,3, Emanuele Rodolà1 1Sapienza University of Rome 2University of Tübingen 3Tübingen AI Center santilli@di.uniroma1.it Abstract Autoregressive decoding limits the efficiency of transformers for Machine Translation (MT). The community proposed specific network architectures and learning-based methods to solve this issue, which are expensive and re- quire changes to the MT model, trading infer- ence speed at the cost of the translation quality. In this paper, we propose to address the prob- lem from the point of view of decoding algo- rithms, as a less explored but rather compelling direction. We propose to reframe the standard greedy autoregressive decoding of MT with a parallel formulation leveraging Jacobi and Gauss-Seidel fixed-point iteration methods for fast inference. This formulation allows to speed up existing models without training or modifi- cations while retaining translation quality. We present three parallel decoding algorithms and test them on different languages and models showing how the parallelization introduces a speedup up to 38% w.r.t. the standard autore- gressive decoding and nearly 2x when scaling the method on parallel resources. Finally, we introduce a decoding dependency graph visual- izer (DDGviz) that let us see how the model has learned the conditional dependence between to- kens and inspect the decoding procedure. 1 Introduction In recent years there have been dramatic improve- ments in Machine Translation (MT) (Edunov et al., 2018; Liu et al., 2020) thanks to the transition to neural models and the advent of the Transformer ar- chitecture (Vaswani et al., 2017). These models can produce high-quality translations while being ex- tremely parallelizable during training. However, Transformers are used sequentially at inference time, generating one token per time (i.e., sending each token as input for the next autoregressive itera- tion). This process of autoregressive inference ham- pers the efficiency of neural machine translation systems in terms of latency, limiting applications and portability. Considering that these systems are extensively used in production multiple times to produce new translations (e.g., Google Translate1, DeepL Translator2), even a minor speedup would be beneficial in the long run, especially if the trans- lation is done on embedded devices. To address this issue, the community proposed ad-hoc trained models specific for parallel ma- chine translation under the umbrella term of Non- Autoregressive Machine Translation models (NAT) (Gu et al., 2018). These models produce the trans- lation in parallel but require (i) a complete reengi- neering of the MT system, (ii) extensive training resources and (iii) complex design choices like dis- tillation from larger autoregressive models. These requirements are quite demanding and not easily satisfiable. For example, production systems are heavily optimized for hardware and software and even introducing a minimal modification requires non-trivial human effort (Wu et al., 2016; Kim et al., 2019). Furthermore, training a new model from scratch is …

Content truncated to 500 words