ERNIE-Doc: A Retrospective Long-Document Modeling Transformer

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 2914–2927 August 1–6, 2021. ©2021 Association for Computational Linguistics 2914 ERNIE-DOC: A Retrospective Long-Document Modeling Transformer Siyu Ding∗, Junyuan Shang∗, Shuohuan Wang, Yu Sun, Hao Tian, Hua Wu and Haifeng Wang Baidu Inc., China {dingsiyu, shangjunyuan, wangshuohuan, sunyu02, tianhao, wu hua, wanghaifeng}@baidu.com Abstract Transformers are not suited for process- ing long documents, due to their quadrati- cally increasing memory and time consump- tion. Simply truncating a long document or applying the sparse attention mechanism will incur the context fragmentation problem or lead to an inferior modeling capability against comparable model sizes. In this paper, we propose ERNIE-DOC, a document-level lan- guage pretraining model based on Recurrence Transformers (Dai et al., 2019). Two well- designed techniques, namely the retrospective feed mechanism and the enhanced recurrence mechanism, enable ERNIE-DOC 1, which has a much longer effective context length, to capture the contextual information of a com- plete document. We pretrain ERNIE-DOC to explicitly learn the relationships among segments with an additional document-aware segment-reordering objective. Various exper- iments were conducted on both English and Chinese document-level tasks. ERNIE-DOC improved the state-of-the-art language model- ing result of perplexity to 16.8 on WikiText- 103. Moreover, it outperformed competitive pretraining models by a large margin on most language understanding tasks, such as text classification and question answering. 1 Introduction Transformers (Vaswani et al., 2017) have achieved remarkable improvements in a wide range of nat- ural language tasks, including language model- ing (Dai et al., 2019), text classification (Yang et al., 2019), and question answering (Devlin et al., 2018; Radford et al., 2019). This success is largely due to the self-attention mechanism, which enables the network to capture contextual information from the *indicates equal contribution. at1Source code and pre-trained checkpoints can be found https://github.com/PaddlePaddle/ERNIE/ tree/repro/ernie-doc. S1 S3 S1 S2 (c) ERNIE-DOC S1 S2 S2 Recurrence Transformer S1 S2 Vanilla or Sparse Transformer P( y | S2 ) P( y | S1 , S2 ) P( y | S1 , S2 , S3 ) ���� (b) S3 S3 Layer-1 Layer-2 Layer-1 Layer-2 Figure 1: Available contextual information utilized by Transformer variants, where a long document D is par- titioned into three segments Si(i ∈ [1, 2, 3]). When training on S2, (a) and (b) optimize the pretraining ob- jective depending only on the contextual information from the current segment or segments in the forward pass, whereas ERNIE-DOC utilizes the contextual in- formation of the entire document for each segment. entire input sequence. Nevertheless, the memory usage and computation complexity caused by the self-attention mechanism grows quadratically with the sequence length, incurring excessive cost when processing a long document on existing hardware. Currently, the most prominent pretrained mod- els, such as BERT (Devlin et al., 2018), are used on fixed-length input segments of a maximum of 512 tokens owing to the aforementioned limita- tion. Thus, a long document input must be parti- tioned into smaller segments …

Content truncated to 500 words