A Dataset for Hyper-Relational Extraction and a Cube-Filling Approach

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1044 - 1058 December 7-11, 2022 ©2022 Association for Computational Linguistics ELMER: A Non-Autoregressive Pre-trained Language Model for Efficient and Effective Text Generation Junyi Li1,3,4, Tianyi Tang1, Wayne Xin Zhao1,4∗, Jian-Yun Nie3 and Ji-Rong Wen1,2,4 1Gaoling School of Artificial Intelligence, Renmin University of China 2School of Information, Renmin University of China 3DIRO, Université de Montréal 4Beijing Key Laboratory of Big Data Management and Analysis Methods {lijunyi,steven_tang}@ruc.edu.cn batmanfly@gmail.com Abstract We study the text generation task under the ap- proach of pre-trained language models (PLMs). Typically, an auto-regressive (AR) method is adopted for generating texts in a token-by- token manner. Despite many advantages of AR generation, it usually suffers from inefficient in- ference. Therefore, non-autoregressive (NAR) models are proposed to generate all target to- kens simultaneously. However, NAR models usually generate texts of lower quality due to the absence of token dependency in the out- put text. In this paper, we propose ELMER: an Efficient and effective PLM for NAR tExt geneRation to explicitly model the token de- pendency during NAR generation. By leverag- ing the early exit technique, ELMER enables the token generations at different layers, ac- cording to their prediction confidence (a more confident token will exit at a lower layer). Be- sides, we propose a novel pre-training objec- tive, Layer Permutation Language Modeling, to pre-train ELMER by permuting the exit layer for each token in sequences. Experiments on three text generation tasks show that ELMER significantly outperforms NAR models and fur- ther narrows the performance gap with AR PLMs (e.g., ELMER (29.92) vs BART (30.61) ROUGE-L in XSUM) while achieving over 10 times inference speedup. 1 Introduction Since the advant of GPT-2 (Radford et al., 2019), pre-trained language models (PLMs) have achieved state-of-the-art performance across text generation tasks, which aim to generate human-like texts on demand (Brown et al., 2020; Li et al., 2022c). These PLMs usually adopt an auto-regressive (AR) fashion to generate texts token-by-token: the next token is predicted based on all previously gener- ated tokens. A major limitation of this approach is that it is hard to be parallelized for the inference ∗Corresponding author process, thus leading to a relatively high inference latency (Gu et al., 2018). Such a limitation prevents AR models from wide deployment in online real- time applications, such as query rewriting in search engines and online chat-bot. Moreover, AR models are prone to suffering from the exposure bias prob- lem since there is a gap between AR training and in- ference (Zeng and Nie, 2021). These concerns have sparked extensive interests in non-autoregressive (NAR) models for text generation (Gu et al., 2018). Compared to AR models, NAR models predict target tokens in all positions simultaneously and independently (Gu et al., 2018). This full paral- lelism leads to an efficient and low-latency infer- ence process. However, the independence assump- tion prevents NAR models from learning the depen- dency among target tokens, resulting in accuracy degradation (Zhan et al., 2022). One …

Content truncated to 500 words