ProphetNet: Predicting Future N-gram for Sequence-to-SequencePre-training

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2401–2410 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 2401 ProphetNet: Predicting Future N-gram for Sequence-to-Sequence Pre-training Weizhen Qi1 ∗†, Yu Yan2†, Yeyun Gong3†, Dayiheng Liu4†, Nan Duan3, Jiusheng Chen2, Ruofei Zhang2, Ming Zhou3 1University of Science and Technology of China, 2Microsoft, 3Microsoft Research Asia, 4Sichuan University 1weizhen@microsoft.com, 2{yyua, jiuchen, bzhang}@microsoft.com 3{yegong, nanduan, mingzhou}@microsoft.com, 4losinuris@gmail.com Abstract This paper presents a new sequence-to- sequence pre-training model called Prophet- Net, which introduces a novel self-supervised objective named future n-gram prediction and the proposed n-stream self-attention mech- anism. Instead of optimizing one-step- ahead prediction in the traditional sequence- to-sequence model, the ProphetNet is opti- mized by n-step ahead prediction that pre- dicts the next n tokens simultaneously based on previous context tokens at each time step. The future n-gram prediction explicitly encour- ages the model to plan for the future tokens and prevent overfitting on strong local cor- relations. We pre-train ProphetNet using a base scale dataset (16GB) and a large-scale dataset (160GB), respectively. Then we con- duct experiments on CNN/DailyMail, Giga- word, and SQuAD 1.1 benchmarks for abstrac- tive summarization and question generation tasks. Experimental results show that Prophet- Net achieves new state-of-the-art results on all these datasets compared to the models using the same scale pre-training corpus. 1 Introduction Large-scale pre-trained language models (Devlin et al., 2018; Radford et al., 2019; Yang et al., 2019) and sequence-to-sequence models (Lewis et al., 2019; Song et al., 2019; Raffel et al., 2019) have achieved remarkable success in downstream tasks. Autoregressive (AR) language modeling, which estimates the probability distribution of the text corpus, is widely used for sequence model- ing and sequence-to-sequence (Seq2Seq) learn- ing (Sutskever et al., 2014). Recently, it also be- comes one of the successful self-supervised objec- tives for large-scale pre-training as used in GPT- Asia.∗Work is done during internship at Microsoft Research † Equal contribution 2 (Radford et al., 2019). Specifically, given a text sequence x = (x1, . . . , xT ), AR language modeling factorizes the likelihood into a product p(x) = �T t=1 p(xt|x<t). In this manner, language models (LMs) and Seq2Seq models are usually trained by teacher forcing. The models are opti- mized to predict the next token given all previous context tokens at each time step. However, as discussed in previous works (Pas- canu et al., 2013; Gulcehre et al., 2017; Serdyuk et al., 2018), AR-based models may prefer to fo- cus on the latest tokens rather than capture long- term dependencies for the next token prediction. The reasons are as follows: (a) Local correlations such as bigram combination are usually stronger than long-term dependencies. (b) Teacher forcing, where the model focus on one-step-ahead predic- tion for each time step, has no explicit bias toward future token planning and modeling. As a result, the model may learn a bias for language modeling; that is, the local token combinations’ modeling is overfitting, but the global coherence and long-term dependency are underfitting (Krueger et …

Content truncated to 500 words