JointGT: Graph-Text Joint Representation Learning for Text Generation from Knowledge Graphs

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 2526–2538 August 1–6, 2021. ©2021 Association for Computational Linguistics 2526 JointGT: Graph-Text Joint Representation Learning for Text Generation from Knowledge Graphs Pei Ke1, Haozhe Ji1, Yu Ran2, Xin Cui2, Liwei Wang3, Linfeng Song4, Xiaoyan Zhu1, Minlie Huang1∗ 1The CoAI group, Department of Computer Science and Technology, Institute for Artificial Intelligence, State Key Lab of Intelligent Technology and Systems, Beijing National Research Center for Information Science and Technology, Tsinghua University, Beijing 100084, China 2Sogou Inc., Beijing, China 3The Chinese University of Hong Kong 4Tencent AI Lab {kp17,jhz20}@mails.tsinghua.edu.cn, {zxy-dcs,aihuang}@tsinghua.edu.cn Abstract Existing pre-trained models for knowledge- graph-to-text (KG-to-text) generation simply fine-tune text-to-text pre-trained models such as BART or T5 on KG-to-text datasets, which largely ignore the graph structure dur- ing encoding and lack elaborate pre-training tasks to explicitly model graph-text align- ments. To tackle these problems, we propose a graph-text joint representation learning model called JointGT. During encoding, we devise a structure-aware semantic aggregation mod- ule which is plugged into each Transformer layer to preserve the graph structure. Fur- thermore, we propose three new pre-training tasks to explicitly enhance the graph-text align- ment including respective text / graph recon- struction, and graph-text alignment in the em- bedding space via Optimal Transport. Exper- iments show that JointGT obtains new state- of-the-art performance on various KG-to-text datasets1. 1 Introduction Knowledge-graph-to-text (KG-to-text) generation aims to generate high-quality texts which are con- sistent with input graphs (Gardent et al., 2017). This task requires to simultaneously encode the graph structure and the content, and effectively leverage the input graphs in the decoding process (Zhao et al., 2020). As a major natural language generation (NLG) task that connects knowledge graphs and texts, this task can further promote the applicability of knowledge graphs in more realis- tic NLG scenarios, such as knowledge-grounded dialogue generation (Zhou et al., 2018a) and story generation (Guan et al., 2019; Ji et al., 2020). Due to the limited amount of graph-text paral- lel data, it’s hard for typical neural text generation ∗ Corresponding author 1The data, codes, and model parameters are available at https://github.com/thu-coai/JointGT. models to learn the alignments between source enti- ties / relations and target tokens from scratch (Guo et al., 2020; Fu et al., 2020). Recent work resorts to constructing general-purpose pre-trained language models for KG-to-text generation. The most com- mon and simple way is to linearize input graphs into text sequences, and directly fine-tune text- to-text Transformer-based pre-trained models like GPT (Radford et al., 2018, 2019), BART (Lewis et al., 2020) or T5 (Raffel et al., 2020) on KG-to- text datasets (Ribeiro et al., 2020a; Kale and Ras- togi, 2020). Benefiting from self-supervised pre- training on large-scale unlabelled text corpora, pre- trained language models can generate high-quality texts via simply fine-tuning, and outperform other models with sophisticated structures. Despite the superior performance of fine-tuning pre-trained models on KG-to-text datasets, we ar- gue that building pre-trained models for KG-to- text generation still faces two major challenges: 1) Structural information loss during encoding. …

Content truncated to 500 words