Self-supervised Graph Masking Pre-training for Graph-to-Text Generation

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 4845 - 4853 December 7-11, 2022 ©2022 Association for Computational Linguistics Self-supervised Graph Masking Pre-training for Graph-to-Text Generation Jiuzhou Han and Ehsan Shareghi Department of Data Science & AI, Monash University {jiuzhou.han, ehsan.shareghi}@monash.edu Abstract Large-scale pre-trained language mod- els (PLMs) have advanced Graph-to- Text (G2T) generation by processing the linearised version of a graph. However, the linearisation is known to ignore the structural information. Additionally, PLMs are typically pre-trained on free text which introduces domain mismatch between pre-training and downstream G2T generation tasks. To address these shortcomings, we propose graph masking pre-training strategies that neither require supervision signals nor adjust the architecture of the underlying pre-trained encoder-decoder model. When used with a pre-trained T5, our approach achieves new state-of-the-art results on WebNLG+2020 and EventNarrative G2T generation datasets. Our method also shows to be very effective in the low-resource setting.1 1 Introduction Graph-to-Text (G2T) generation (Gatt and Krah- mer, 2018) is the task of generating natural lan- guage from graph-structured data. While there are several tasks that could leverage a G2T com- ponent (Zhou et al., 2018; Ji et al., 2020; Chen et al., 2021) the direct generation of text descrip- tion from knowledge graphs (KGs) have attracted a lot of attention due to its potential in providing a more accessible presentation of knowledge to non-experts (Schmitt et al., 2020). In parallel, Transformer-based (Vaswani et al., 2017) pre-trained language models (PLMs) such as BART (Lewis et al., 2019), and T5 (Raffel et al., 2019) have facilitated state-of-the-art (SotA) re- sults on several tasks, including earlier SotA results for G2T (Ribeiro et al., 2020; Kale and Rastogi, 2020; Mager et al., 2020). It has been argued that their success, in part, is due to factual memorisa- tion that guides the generation (Ribeiro et al., 2020). 1Our code is available at https://github.com/ Jiuzhouh/Graph-Masking-Pre-training. Although PLMs benefit the G2T generation, the lin- earisation step required to use these models ignores the structural information of the graph (Wang et al., 2021), while explicitly modelling structured data could also lead to catastrophic forgetting of distri- butional knowledge (Ribeiro et al., 2021). To address this, Wang et al. (2021) proposed adding extra positional embedding layers to cap- ture the inter-dependency structures of input graphs. Ribeiro et al. (2021) proposed using a structure- aware adapter in PLMs to supplement the input with its graph structure. For table data, Xing and Wan (2021) considered the structure of the table input by predicting the surrounding cells for a cell in a table. However, these methods either change the design of the PLMs (limiting their use for other task settings) or require labelled training data to capture the graph structure information. In this work, we propose self-supervised graph masking pre-training strategies to enhance the structure awareness of PLMs. To achieve this, we formulate several graph masking strategies to inject local and global awareness of the input structure into the PLM. Our method has two key advantages: (i) it …

Content truncated to 500 words