KGPT: Knowledge-Grounded Pre-Training for Data-to-Text Generation

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 8635–8648, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 8635 KGPT: Knowledge-Grounded Pre-Training for Data-to-Text Generation Wenhu Chen, Yu Su, Xifeng Yan, William Yang Wang University of California, Santa Barbara, CA, USA {wenhuchen, xyan, william}@cs.ucsb.edu, su.809@osu.edu Abstract Data-to-text generation has recently attracted substantial interests due to its wide applica- tions. Existing methods have shown impres- sive performance on an array of tasks. How- ever, they rely on a significant amount of la- beled data for each task, which is costly to acquire and thus limits their application to new tasks and domains. In this paper, we propose to leverage pre-training and transfer learning to address this issue. We propose a knowledge-grounded pre-training (KGPT), which consists of two parts, 1) a general knowledge-grounded generation model to gen- erate knowledge-enriched text. 2) a pre- training paradigm on a massive knowledge- grounded text corpus crawled from the web. The pre-trained model can be fine-tuned on various data-to-text generation tasks to gener- ate task-specific text. We adopt three settings, namely fully-supervised, zero-shot, few-shot to evaluate its effectiveness. Under the fully- supervised setting, our model can achieve re- markable gains over the known baselines. Un- der zero-shot setting, our model without see- ing any examples achieves over 30 ROUGE-L on WebNLG while all other baselines fail. Un- der the few-shot setting, our model only needs about one-fifteenth as many labeled examples to achieve the same level of performance as baseline models. These experiments consis- tently prove the strong generalization ability of our proposed framework1. 1 Introduction Data-to-text generation, i.e., generating textual de- scription from structured data, is an important task with many real-world applications such as gener- ating weather reports (Liang et al., 2009), sports news (Wiseman et al., 2017), dialog response (Wen et al., 2016; Duˇsek et al., 2019), etc. Neural gener- 1https://github.com/wenhuchen/KGPT Moses Malone, Hakeem Olajuwon, and James Harden have been named the NBA's Most Valuable Player while playing for the Rockets, for a total of four MVP awards. Moses Houston Rockets MVP human Houston Rockets NBA plays for awarded instance of member of Power Forward position male gender award instance of Basketball sport s 1956 inception Moses;Harden;Hakeem Description:,American,Basketball,Player Ground Language Model Hakeem Harden awarded awarded Figure 1: An example from the constructed KGTEXT, which pairs a hyperlinked sentence from Wikipedia with a knowledge subgraph from WikiData. ation models based on different strategies like soft- template (Wiseman et al., 2018; Ye et al., 2020), copy-mechanism (See et al., 2017), content plan- ning (Reed et al., 2018; Moryossef et al., 2019), and structure awareness (Liu et al., 2018; Colin and Gardent, 2019) have achieved impressive results. However, existing studies are primarily focused on fully supervised setting requiring substantial labeled annotated data for each subtask, which re- stricts their adoption in real-world applications. In this paper, we are interested in developing a general-purpose model that can easily adapt to different domains/tasks and achieve strong perfor- mance with only a small amount or …

Content truncated to 500 words