Structural Adapters in Pretrained Language Models for AMR-to-Text Generation

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4269–4282 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 4269 Structural Adapters in Pretrained Language Models for AMR-to-Text Generation Leonardo F. R. Ribeiro†, Yue Zhang‡ and Iryna Gurevych† †Ubiquitous Knowledge Processing Lab, Technical University of Darmstadt ‡School of Engineering, Westlake University ribeiro@aiphes.tu-darmstadt.de Abstract Pretrained language models (PLM) have recently advanced graph-to-text generation, where the input graph is linearized into a se- quence and fed into the PLM to obtain its representation. However, efficiently encoding the graph structure in PLMs is challenging because such models were pretrained on nat- ural language, and modeling structured data may lead to catastrophic forgetting of distri- butional knowledge. In this paper, we pro- pose STRUCTADAPT, an adapter method to en- code graph structure into PLMs. Contrary to prior work, STRUCTADAPT effectively mod- els interactions among the nodes based on the graph connectivity, only training graph structure-aware adapter parameters. In this way, we incorporate task-specific knowledge while maintaining the topological structure of the graph. We empirically show the benefits of explicitly encoding graph structure into PLMs using STRUCTADAPT, outperforming the state of the art on two AMR-to-text datasets, train- ing only 5.1% of the PLM parameters.1 1 Introduction Data-to-text tasks aim to generate meaningful and coherent natural language text that faithfully con- veys structured data. Some examples of structured information include tables (Parikh et al., 2020), Knowledge Graphs (KGs) (Gardent et al., 2017; Vougiouklis et al., 2018) and Abstract Meaning Representation (AMR) (Banarescu et al., 2013). In this work, we focus on AMR-to-text generation where the goal is to generate a fluent and gram- matical sentence that is faithful to a given AMR graph (See Figure 1a). AMR is a semantic for- malism that has received much research interest (Song et al., 2018; Guo et al., 2019; Ribeiro et al., 2019; Opitz et al., 2020, 2021; Fu et al., 2021) and has been shown to benefit downstream tasks 1Our code and checkpoints are available at https://github.com/UKPLab/StructAdapt. power-01 (a) Pretrained Linearized Model power :ARG1 she :mod more :purpose achieve :ARG0 she power :ARG1 she :mod more :purpose achieve :ARG0 she more achieve-01 :purpose :mod she :ARG1 :ARG0 (b) (c) Fine-tuning only with graph linearization More power to her to achieve. Lightweight fine-tuning with graph structure Pretrained Model with StructAdapt More power to her for her achievements. AMR graph Figure 1: (a) AMR for the sentence More power to her for her achievements. While in (b) the pretrained model gets as input the graph linearization, in (c) it addition- ally receives the graph connectivity information. such as text summarization (Liao et al., 2018) and machine translation (Song et al., 2019). Both sta- tistical (Flanigan et al., 2016; Pourdamghani et al., 2016) and neural methods (Bai et al., 2020; Cai and Lam, 2020) have been investigated for AMR- to-text generation, and dominant methods make use of Graph Neural Networks (GNNs) (Kipf and Welling, 2017) or Transformers (Vaswani et al., 2017) for representing the input graph. Pretrained …

Content truncated to 500 words