PRINCE: Prefix-Masked Decoding for Knowledge Enhanced Sequence-to-Sequence Pre-Training

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 2675 - 2681 December 7-11, 2022 ©2022 Association for Computational Linguistics PRINCE: Prefix-Masked Decoding for Knowledge Enhanced Sequence-to-Sequence Pre-Training Song Xu∗, Haoran Li∗, Peng Yuan, Youzheng Wu, Xiaodong He JD AI Research {xusong28, lihaoran24}@jd.com Abstract Pre-trained Language Models (PLMs) have shown effectiveness in various Natural Lan- guage Processing (NLP) tasks. Denoising au- toencoder is one of the most successful pre- training frameworks, learning to recompose the original text given a noise-corrupted one. The existing studies mainly focus on injecting noises into the input. This paper introduces a simple yet effective pre-training paradigm, equipped with a knowledge-enhanced decoder that predicts the next entity token with noises in the prefix, explicitly strengthening the rep- resentation learning of entities that span over multiple input tokens. Specifically, when pre- dicting the next token within an entity, we feed masks into the prefix in place of some of the previous ground-truth tokens that constitute the entity. Our model achieves new state-of-the-art results on two knowledge-driven data-to-text generation tasks with up to 2% BLEU gains. 1 Introduction Pre-trained language models (PLMs), such as BERT (Devlin et al., 2019), MASS (Song et al., 2019), BART (Lewis et al., 2020), and T5 (Raffel et al., 2020), have had remarkable performances in various Natural Language Processing (NLP) tasks thanks to the use of denoising autoencoder pre- training schema that is optimized to reconstruct the original text given a noise-corrupted one. Despite its common usage, to the best of our knowledge, existing work mainly focuses on injecting noises into the encoding sequence, while the feasibility of injecting noises into the decoding sequence for PLMs remains an open question. This is the pri- mary interest of this work. On the other hand, recent researches (Zhang et al., 2019; Peters et al., 2019; Xiong et al., 2020; Wang et al., 2019, 2020; Ke et al., 2020; Tian et al., 2020; Xu et al., 2021) demonstrate that enhanc- ing PLMs with real-world knowledge is crucial ∗Equal contribution. for knowledge-driven downstream tasks, includ- ing entity typing, relation classification, sentiment analysis, and entity-related question answering. However, the pre-training objectives for the above- mentioned studies are usually designed for Natu- ral Language Understanding (NLU) tasks. In this work, we focus on knowledge-oriented sequence- to-sequence (Seq2Seq) pre-training objectives for PLMs, so that they can be applied to Natural Lan- guage Generation (NLG) tasks, such as the data-to- text task. Given a noise-corrupted sentence, such as “Tom Cruise was born in [MASK] [MASK] [MASK] in the year 1962”, the standard Seq2Seq pre-training would predict the masked text fragment (i.e., “New York City”) token by token, while our proposed pre- training would generate the correct output by learn- ing facts regarding the concerned entity. We argue that given merely “Tom Cruise was born in”, if a PLM can immediately predict the masked fragment to be “New York City”, it suggests that the PLM has learned the fact (“Tom Cruise”, “born in”, “New York City”). In contrast, …

Content truncated to 500 words