Towards More Efficient Insertion Transformer with Fractional Positional Encoding

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 1564–1572 May 2-6, 2023 ©2023 Association for Computational Linguistics Towards More Efficient Insertion Transformer with Fractional Positional Encoding Zhisong Zhang1∗, Yizhe Zhang2†, Bill Dolan3 1Carnegie Mellon University, 2Apple Inc., 3Microsoft Research zhisongz@cs.cmu.edu,yizhe.zhang@hotmail.com,billdol@microsoft.com Abstract Auto-regressive neural sequence models have been shown to be effective across text genera- tion tasks. However, their left-to-right decod- ing order prevents generation from being par- allelized. Insertion Transformer (Stern et al., 2019) is an attractive alternative that allows outputting multiple tokens in a single gener- ation step. Nevertheless, due to the incom- patibility between absolute positional encod- ing and insertion-based generation schemes, it needs to refresh the encoding of every token in the generated partial hypothesis at each step, which could be costly. We design a novel reusable positional encoding scheme for Inser- tion Transformers called Fractional Positional Encoding (FPE), which allows reusing repre- sentations calculated in previous steps. Em- pirical studies on various text generation tasks demonstrate the effectiveness of FPE, which leads to floating-point operation reduction and latency improvements on batched decoding. 1 Introduction Transformer-based models (Vaswani et al., 2017) have been successfully applied to various text gen- eration tasks (Gong et al., 2019; Wang et al., 2019; Ahmad et al., 2020; Zhang et al., 2020a; Lewis et al., 2020; Brown et al., 2020). Most of these models utilize a fixed left-to-right auto-regressive generation strategy, where the strict factorization means that the model can only generate one token per step. This makes it difficult to parallelize the decoding process, while parallel generation may help to improve decoding efficiency. Recently, insertion-based sequence-generation models (Stern et al., 2019; Gu et al., 2019a; Welleck et al., 2019) have been developed as attractive alter- natives to the auto-regressive ones by allowing flex- ible generation order. In particular, the Insertion Transformer (Stern et al., 2019), which combines ∗Work done during an internship at Microsoft Research. †Work done at Microsoft Research. the Transformer architecture and the insertion- based strategy, can match the performance of an auto-regressive model while requiring many fewer decoding steps with parallel generation. The original Insertion Transformer utilizes ab- solute positional encoding as in the vanilla auto- regressive transformer. In the vanilla transformer, due to its left-to-right generation scheme, tokens’ absolute positions do not change; thus, previous computation can be reused. However, this property no longer holds if insertion is allowed, and the In- sertion Transformer re-encodes all previously gen- erated tokens at each decoding step, which brings additional computational overheads. In this work, we propose a reusable positional en- coding scheme called Fractional Positional Encod- ing (FPE) to accelerate the Insertion Transformer. This scheme dynamically calculates each token’s positional representations according to its left and right neighbors at insertion time. In this way, each token’s positional representations will not change during the decoding process so that the computa- tion can be reusable in the same way as the vanilla transformer, leading to a reduction of computation for the Insertion Transformer. …

Content truncated to 500 words