Investigating the Effect of Relative Positional Embeddings on AMR-to-Text Generation with Structural Adapters

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 727–736 May 2-6, 2023 ©2023 Association for Computational Linguistics Investigating the Effect of Relative Positional Embeddings on AMR-to-Text Generation with Structural Adapters Sebastien Montella∗ Orange Innovation / Lannion, France Aix-Marseille Univ. CNRS, LIS / Marseille, France sebastien.montella@huawei.com Alexis Nasr AMU/CNRS/LIS, Marseille, France alexis.nasr@lis-lab.fr Johannes Heinecke Orange Innovation / Lannion, France johannes.heinecke@orange.com Frederic Bechet AMU/CNRS/LIS, Marseille, France frederic.bechet@lis-lab.fr Lina M. Rojas-Barahona Orange Innovation / Lannion, France linamaria.rojasbarahona @orange.com Abstract Text generation from Abstract Meaning Repre- sentation (AMR) has substantially benefited from the popularized Pretrained Language Models (PLMs). Myriad approaches have lin- earized the input graph as a sequence of to- kens to fit the PLM tokenization requirements. Nevertheless, this transformation jeopardizes the structural integrity of the graph and is therefore detrimental to its resulting represen- tation. To overcome this issue, Ribeiro et al. (2021b) have recently proposed StructAdapt, a structure-aware adapter which injects the input graph connectivity within PLMs using Graph Neural Networks (GNNs). In this paper, we investigate the influence of Relative Position Embeddings (RPE) on AMR-to-Text, and, in parallel, we examine the robustness of Struc- tAdapt. Through ablation studies, graph attack and link prediction, we reveal that RPE might be partially encoding input graphs. We sug- gest further research regarding the role of RPE will provide valuable insights for Graph-to-Text generation. 1 Introduction Earliest works on AMR-to-Text generation were mostly based on statistical methods. A common practice was to convert AMR-to-Text task into an already studied problems such as Tree-to-Text ∗Work done while at Orange Innovation (Lannion, France). Now at Huawei Edinburgh Research Centre (United King- dom). (Flanigan et al., 2016; Lampouras and Vlachos, 2017), aligned text-to-text (Pourdamghani et al., 2016), Travel Sales Problems (Song et al., 2016) or Grammatical Framework (Ranta, 2011). Recently, most methods are neural-centered with an encoder- decoder architecture (Sutskever et al., 2014) as a backbone (Konstas et al., 2017; Takase et al., 2016; Cao and Clark, 2019). Unfortunately, this archi- tecture coerces the AMR to be linearized as a se- quence of tokens. This ends up in structural infor- mation loss. To tackle this issue, several strategies have attempted to integrate structure using message propagation (Song et al., 2018; Guo et al., 2019; Damonte and Cohen, 2019; Ribeiro et al., 2019; Zhang et al., 2020b; Zhao et al., 2020). A lim- itation of those is the absence of pretraining, as demonstrated by Ribeiro et al. (2021a). To this end, Ribeiro et al. (2021b) introduced StructAdapt for lightweight AMR-to-Text with structural adapters. As linearization and tokenization of the input graph are mandatory steps for PLMs, StructAdapt first defines a new graph where nodes are the result- ing subwords from the tokenization. As a result, adapter can henceforth include GNN layers oper- ating on the subsequent graph while leveraging pretrained representations. However, although studies have been made to probe position embeddings (Wang and Chen, 2020; Wang et al., 2021; Dufter et al., 2022), their role on graph encoding has remained …

Content truncated to 500 words