Weisfeiler-Leman in the Bamboo: Novel AMR Graph Metrics and a Benchmark for AMR Graph Similarity

Weisfeiler-Leman in the BAMBOO : Novel AMR Graph Metrics and a Benchmark for AMR Graph Similarity Juri Opitz1 Angel Daza2 Anette Frank1 1Dept. of Computational Linguistics, Heidelberg University, Germany 2CLTL, Vrije Universiteit Amsterdam, The Netherlands {opitz, frank}@cl.uni-heidelberg.de, j.a.dazaarevalo@vu.nl Abstract Several metrics have been proposed for as- sessing the similarity of (abstract) meaning representations (AMRs), but little is known about how they relate to human similarity rat- ings. Moreover, the current metrics have com- plementary strengths and weaknesses: Some emphasize speed, while others make the align- ment of graph structures explicit, at the price of a costly alignment step. In this work we propose new Weisfeiler-Leman AMR similarity metrics that unify the strengths of previousmetrics, while mitigating their weak- nesses. Specifically, our new metrics are able to match contextualized substructures and in- duce n:m alignments between their nodes. Fur- thermore, we introduce a Benchmark for AMR Metrics based on Overt Objectives (BAMBOO ), the first benchmark to support empirical as- sessment of graph-based MR similarity met- rics. BAMBOO maximizes the interpretability of results by defining multiple overt objectives that range from sentence similarity objectives to stress tests that probe a metric’s robust- ness against meaning-altering and meaning- preserving graph transformations. We show the benefits of BAMBOO by profiling previous metrics and our own metrics. Results indicate that our novel metrics may serve as a strong baseline for future work. 1 Introduction Meaning representations aim at capturing the meaning of text in an explicit graph format. A prominent framework is abstract meaning repre- sentation (AMR), proposed by Banarescu et al. (2013). AMR views sentences as rooted, directed, acyclic, labeled graphs. Their nodes are variables, attributes, or (open-class) concepts and are con- nected with edges that express semantic relations. There are many use cases in which we need to compare or relate two AMR graphs. A common situation is found in parser evaluation, where AMR metrics are widely applied (May, 2016; May and Priyadarshi, 2017).1 Yet, there are more situations where we need to measure similarity of meaning as expressed in AMR graphs. For example, Bonial et al. (2020) leverage AMR met- rics in a semantic search engine for COVID-19 queries, Naseem et al. (2019) use metric feedback to reinforce AMR parsers, Opitz (2020) emulates metrics for referenceless AMR ranking and rating, and Opitz and Frank (2021) use AMR metrics for NLG evaluation. So far, multiple AMR metrics (Cai and Knight, 2013; Cai and Lam, 2019; Song and Gildea, 2019; Anchiˆeta et al., 2019; Opitz et al., 2020) have been proposed to assess AMR similarity. How- ever, due to a lack of an appropriate evaluation benchmark, we have no empirical evidence that could tell us more about their strengths and weak- nesses or offer insight about which metrics may be preferable over others in specific use cases. Additionally, we would like to move beyond the aforementioned metrics and develop new metrics that account for graded similarity of graph sub- structures, which is not an easy task. However, it is crucial when we need …

Content truncated to 500 words