Promoting Graph Awareness in Linearized Graph-to-Text Generation

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 944–956 August 1–6, 2021. ©2021 Association for Computational Linguistics 944 Promoting Graph Awareness in Linearized Graph-to-Text Generation Alexander Hoyle♣∗ Ana Marasovi´c†3 Noah A. Smith†3 ♣Department of Computer Science, University of Maryland, College Park †Allen Institute for Artificial Intelligence 3Paul G. Allen School of Computer Science and Engineering, University of Washington hoyle@umd.edu, {anam,noah}@allenai.org Abstract Generating text from structured inputs, such as meaning representations or RDF triples, has often involved the use of specialized graph- encoding neural networks. However, recent applications of pretrained transformers to lin- earizations of graph inputs have yielded state- of-the-art generation results on graph-to-text tasks. Here, we explore the ability of these linearized models to encode local graph struc- tures, in particular their invariance to the graph linearization strategy and their ability to re- construct corrupted inputs. Our findings mo- tivate solutions to enrich the quality of mod- els’ implicit graph encodings via scaffolding. Namely, we use graph-denoising objectives implemented in a multi-task text-to-text frame- work. We find that these denoising scaffolds lead to substantial improvements in down- stream generation in low-resource settings. 1 Introduction Parameter-rich pretrained transformer language models succeed at generating text that is prima facie fluent, but that closer inspection will often reveal to be semantically transgressive (Bisk et al., 2020). Indeed, there is limited practical use for unconditional text generation: we expect language to relate to some identifiable, extrinsic meaning. When a system communicates information to an individual in natural language, it will typically rely on a structured representation of that information. Consequently, generating text that faithfully conveys structured data is an important goal in NLP, where inputs can take the form of tables (ToTTo, Parikh et al., 2020), RDF triples (e.g., WebNLG, Gardent et al., 2017), or Abstract Meaning Representations (AMR, Flanigan et al., 2016). NLP datasets in this domain consists of pairs of structured data (e.g., <henri_matisse, ∗Work undertaken during an internship at AI2. want :arg0 :arg1 :arg0 boy go (1) Linearize graph (2) Finetune with one linearization The boy wants to go Pretrained Language Model (want :arg0 boy :arg1 (go :arg0 boy)) (go :arg0 (boy :arg0-of want) :arg1-of want) (want :arg0 To go the boy wants Finetuned Language Model (go :arg0 (3) Evaluate with an alternative linearization Figure 1: Diagram of our adversarial evaluation proce- dure for graph-to-text generation using pretrained lan- guage models (§3.2). (1) A graph can admit multiple possible linearizations. (2) Following standard practice, we train with a single linearization. (3) At evaluation time, we present the model with a meaning-preserving alternative. hasOccupation, artist>) and a represen- tation of that data in text (“Matisse is an artist.”). Importantly, these types of inputs can be encoded as graphs. Accordingly, advances in neural archi- tectures designed to explicitly encode graphs, such as graph neural networks (GNNs, Kipf and Welling, 2017) and graph transformers, have been used in these graph-to-text settings (Zhu et al., 2019; Zhao et al., 2020; Wang et al., 2020, to name a few). But graphs can also …

Content truncated to 500 words