Dynamic and Efficient Inference for Text Generation via BERT Family

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 2883–2897 July 9-14, 2023 ©2023 Association for Computational Linguistics Dynamic and Efficient Inference for Text Generation via BERT Family Xiaobo Liang1 Juntao Li1∗ Lijun Wu2 Ziqiang Cao1 Min Zhang1 1Soochow University, 2Microsoft Research xbliang3@stu.suda.edu.cn, {ljt,zqcao,minzhang}@suda.edu.cn lijuwu@microsoft.com Abstract Despite the excellent performance of Pre- trained Language Models on many text genera- tion tasks, they suffer from inefficient inference on computation and memory due to their large- scale parameters and the universal autoregres- sive decoding paradigm. In this work, we pro- pose a novel fine-tuning method DEER, which can make a single pre-trained model support Dynamic and Efficient infERence and achieve an adaptive trade-off between model perfor- mance and latency. In particular, our critical in- sight is to jointly utilize the non-autoregressive (NAR) generation and dynamic parameter prun- ing techniques, which can flexibly control the decoding iteration steps and model sizes ac- cording to memory and latency limitations. Be- sides, we also explore the effectiveness of the pre-trained MLMs (i.e., the BERT family) for text generation tasks since their bidirectional attention nature is more suitable for the NAR training objective. Extensive experiments on both monolingual and multilingual pre-trained MLMs demonstrate the effectiveness of our pro- posed DEER method by consistently achieving (1) higher BLEU scores than the strong autore- gressive Transformer model on three neural machine translation tasks with 3 → 12 times speedup, (2) competitive performance (but with much faster inference speed) compared with the BART model on four GLGE benchmark tasks. Our code will be publicly available at GitHub1. 1 Introduction Large-scale pre-trained language models (Devlin et al., 2019; Radford et al., 2019; Brown et al., 2020; Chowdhery et al., 2022) have shown great po- tential in achieving impressive performance; how- ever, they are accompanied by substantial compu- tational complexities and occupy significant mem- ory space. These factors pose obstacles to their practical implementation in real-world applications. ∗Corresponding Author 1https://github.com/dropreg/DEER While recent studies (Sanh et al., 2019; Jiao et al., 2020) have made attempts to address the chal- lenges associated with compressing and acceler- ating inference for pre-trained Transformer models, the majority of these efforts have concentrated on techniques such as knowledge distillation (Song et al., 2020), quantization (Bai et al., 2021; Tao et al., 2022), and parameter pruning (Xia et al., 2022). The pre-trained non-autoregressive gener- ation paradigm has received limited attention and remains relatively unexplored. To fill this blank, we first summarize two main difficulties in the deployment and application of large generative models. Firstly, the prevailing gen- erative models currently employ an autoregressive approach to generate target tokens incrementally, as seen in models like BART (Lewis et al., 2020) and T5 (Raffel et al., 2020). While these models have gained popularity and demonstrated effectiveness, their autoregressive nature hinders efficient infer- ence through parallelization, resulting in inefficien- cies. Secondly, task-specific fine-tuning is crucial when deploying pre-trained models on diverse edge devices (Sun et al., 2020; Xu et al., 2021). It is …

Content truncated to 500 words