Graph Pre-training for AMR Parsing and Generation

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 6001 - 6015 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Graph Pre-training for AMR Parsing and Generation Xuefeng Bai♠♡ , Yulong Chen♠♡ , Yue Zhang♡♢ ♠ Zhejiang University, China ♡ School of Engineering, Westlake University, China ♢ Institute of Advanced Technology, Westlake Institute for Advanced Study, China Abstract Abstract meaning representation (AMR) high- lights the core semantic information of text in a graph structure. Recently, pre-trained lan- guage models (PLMs) have advanced tasks of AMR parsing and AMR-to-text generation, re- spectively. However, PLMs are typically pre- trained on textual data, thus are sub-optimal for modeling structural knowledge. To this end, we investigate graph self-supervised training to improve the structure awareness of PLMs over AMR graphs. In particular, we introduce two graph auto-encoding strategies for graph- to-graph pre-training and four tasks to integrate text and graph information during pre-training. We further design a unified framework to bridge the gap between pre-training and fine-tuning tasks. Experiments on both AMR parsing and AMR-to-text generation show the superiority of our model. To our knowledge, we are the first to consider pre-training on semantic graphs. 1 Introduction Abstract meaning representation (AMR; Banarescu et al. (2013)) is a semantic structure formalism. It represents the meaning of a text in a rooted directed graph, where nodes represent basic semantic units such as entities and predicates, and edges represent their semantic relations, respectively. One example is shown in Figure 1(a), with the corresponding sentence in Figure 1(b). Serving as a structural representation, AMR has been shown useful for NLP tasks such as text summarization (Liu et al., 2015; Liao et al., 2018; Chen et al., 2021), machine translation (Song et al., 2019), information extrac- tion (Huang et al., 2016; Zhang and Ji, 2021) and dialogue systems (Bai et al., 2021). There are two fundamental NLP tasks concern- ing AMR, namely AMR parsing (Flanigan et al., 2014; Konstas et al., 2017; Lyu and Titov, 2018; Guo and Lu, 2018; Zhang et al., 2019a; Cai and Lam, 2020; Bevilacqua et al., 2021) and AMR-to- text generation (Konstas et al., 2017; Song et al., possible :domain go-01 :polarity negative :arg0 boy (a) (b) The boy cannot go. AMR parsing AMR-to-text Figure 1: Illustration of AMR tasks: (a) an AMR graph; (b) a corresponding sentence. 2018; Zhu et al., 2019; Zhao et al., 2020; Bai et al., 2020; Ribeiro et al., 2021a). As shown in Figure 1, the former transforms a textual input (e.g., a sen- tence) into a corresponding AMR structure, and the latter transforms an AMR input into a fluent and grammatical sentence that conveys the same meaning. A common challenge to both tasks is that AMR exists in the form of a graph structure, which is difficult for neural models to learn with limited human-curated data. Recently, large-scale pre-trained sequence-to- sequence (seq2seq) language models (Lewis et al., 2020; Raffel et al., 2020) have been shown use- ful for both tasks above. …

Content truncated to 500 words