GAP: A Graph-aware Language Model Framework for Knowledge Graph-to-Text Generation

Proceedings of the 29th International Conference on Computational Linguistics, pages 5755–5769 October 12–17, 2022. 5755 GAP: A Graph-aware Language Model Framework for Knowledge Graph-to-Text Generation Anthony Colas∗and Mehrdad Alvandipour* and Daisy Zhe Wang Department of Computer Science, University of Florida {acolas1, m.alvandipour, daisyw}@ufl.edu Abstract Recent improvements in KG-to-text genera- tion are due to additional auxiliary pre-training tasks designed to give the fine-tune task a boost in performance. These tasks require extensive computational resources while only suggesting marginal improvements. Here, we demonstrate that by fusing graph-aware elements into exist- ing pre-trained language models, we are able to outperform state-of-the-art models and close the gap imposed by additional pre-training tasks. We do so by proposing a mask struc- ture to capture neighborhood information and a novel type encoder that adds a bias to the graph- attention weights depending on the connection type. Experiments on two KG-to-text bench- mark datasets show our models are competitive while involving fewer parameters and no addi- tional pre-training tasks. By formulating the problem as a framework, we can interchange the various proposed components and begin in- terpreting KG-to-text generative models based on the topological and type information found in a graph. 1 Introduction Due to the amount of data stored in Knowledge Graphs (KGs) (Auer et al., 2007; Vrandeˇci´c and Krötzsch, 2014; Bollacker et al., 2008; Yates et al., 2007; Bodenreider, 2004; Wishart et al., 2018), they are important to properly transcribe into nat- ural language sentences, making them more eas- ily comprehensible to a larger audience. This task, termed KG-to-text, has found recent suc- cess in generating knowledge-grounded dialog re- sponses (Wen et al., 2016; Zhou et al., 2018), ques- tion answering (He et al., 2017; Bhowmik and de Melo, 2018; Pal et al., 2019; Agarwal et al., 2021), story generation (Guan et al., 2019; Ji et al., 2020), and event narration (Colas et al., 2021). KG- to-text involves encoding a KG, often sparse, in ∗These authors contributed equally. The Great Debate, also called the Shapley - Curtis Debate, was held on 26 April 1920 at the National Museum of Natural History, between the astronomers Harlow Shapley and Heber Doust Curtis. The format of the Great Debate has been used subsequently to argue the nature of fundamental questions in astronomy. alternative participant participant location instance of Harlow Shapley Heber Doust Curtis topic Shapley– Curtis Debate National Museum of Natural History Debate 26 April 1920 point in time start time end time Astronomy Great Debate Figure 1: Given a graph, KG-to-text generation aims to describe the entities, relations, and its inherent structure via natural language text (grey callout). Corresponding graph-text components are color-coded. order to generate a coherent and representative tex- tual description of the KG as shown in Figure 1. In contrast, Abstract Meaning Representation (AMR)- to-text deals with a more restrictive space, where graphs follow a predefined dense, connected tem- plate (Ribeiro et al., 2021; Koncel-Kedziorski et al., 2019). Thus, when encoding a KG, one should carefully consider the graph’s structure to properly generate its …

Content truncated to 500 words