Evaluate AMR Graph Similarity via Self-supervised Learning

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 16112–16123 July 9-14, 2023 ©2023 Association for Computational Linguistics Evaluate AMR Graph Similarity via Self-supervised Learning Ziyi Shou and Fangzhen Lin Department of Computer Science and Engineering HKUST-Xiaoi Joint Laboratory The Hong Kong University of Science and Technology {zshou, flin}@cse.ust.hk Abstract In work on AMR (Abstract Meaning Represen- tation), similarity metrics are crucial as they are used to evaluate AMR systems such as AMR parsers. Current AMR metrics are all based on nodes or triples matching without consider- ing the entire structures of AMR graphs. To address this problem, and inspired by learned similarity evaluation on plain text, we propose AMRSim, an automatic AMR graph similarity evaluation metric. To overcome the high cost of collecting human-annotated data, AMRSim automatically generates silver AMR graphs and utilizes self-supervised learning methods. We evaluated AMRSim on various datasets and found that AMRSim significantly improves the correlations with human semantic scores and remains robust under diverse challenges. We also discuss how AMRSim can be extended to multilingual cases.1 1 Introduction An Abstract Meaning Representation (AMR; Ba- narescu et al., 2013) is a rooted, directed graph where nodes represent concepts and edges corre- spond to relations of concepts. The goal of an AMR metric is to evaluate the similarities of pairs of AMR graphs so that it can be used to evaluate the outputs of AMR generators such as parsers. Therefore, a good AMR metric is crucial for the design and evaluation of AMR parsers. However, the research on AMR metrics has so far lagged far behind the work on AMR parsing. Current AMR metrics either transfer AMR graphs to triples and consider the one-to-one match- ing of variables (Cai and Knight, 2013) or linearize AMR graphs as sequences and calculate n-gram matching (Song and Gildea, 2019). These metrics fail to consider the entire AMR structure and lack flexibility, resulting in poor correlation with human annotations. 1The code and datasets can be found at https://github. com/zzshou/AMRSim. Inspired by plain-text automatic similarity as- sessment methods that encode sentences into latent semantic representations to measure a similarity of the two representations (Reimers and Gurevych, 2019), we propose to learn the automatic assess- ment of AMR graph similarity through a similar pipeline. Our proposed metric, called AMRSim, adopts the pre-trained language model BERT as the backbone and incorporates GNN adapters to capture the structural information of AMR graphs. To overcome the high cost of collecting training data, we utilize self-supervised learning methods. The training objective is to maximize the dot prod- uct between positive embeddings and to minimize the dot product between different encodings. In contrast to one-to-one matching metrics, our AMR- Sim is alignment-free by computing the cosine of contextualized token embeddings. The prediction process can be considerably accelerated by lever- aging GPUs. We experiment with AMRSim on the trans- formed STSB (Baudiš et al., 2016) and SICK (Marelli et al., 2014) datasets, which contain pairs of AMR graphs and corresponding …

Content truncated to 500 words