AMR Similarity Metrics from Principles

AMR Similarity Metrics from Principles Juri Opitz and Letitia Parcalabescu and Anette Frank Department for Computational Linguistics Heidelberg University 69120 Heidelberg {opitz,parcalabescu,frank}@cl.uni-heidelberg.de Abstract Different metrics have been proposed to com- pare Abstract Meaning Representation (AMR) graphs. The canonical SMATCH metric (Cai and Knight, 2013) aligns the variables of two graphs and assesses triple matches. The recent SEMBLEU metric (Song and Gildea, 2019) is based on the machine-translation metric BLEU (Papineni et al., 2002) and increases compu- tational efficiency by ablating the variable- alignment. In this paper, i) we establish criteria that enable researchers to perform a principled assessment of metrics comparing meaning rep- resentations like AMR; ii) we undertake a thorough analysis of SMATCH and SEMBLEU where we show that the latter exhibits some undesirable properties. For example, it does not conform to the identity of indiscernibles rule and introduces biases that are hard to control; and iii) we propose a novel metric S2MATCH that is more benevolent to only very slight meaning deviations and targets the fulfilment of all established criteria. We assess its suitability and show its advantages over SMATCH and SEMBLEU. 1 Introduction Proposed in 2013, the aim of Abstract Meaning Representation (AMR) is to represent a sentence’s meaning in a machine-readable graph format (Banarescu et al., 2013). AMR graphs are rooted, acyclic, directed,and edge-labeled. Entities, events, properties, and states are represented as variables that are linked to corresponding concepts (encoded as leaf nodes)via is-instance relations (cf. Figure 1, left). This structure allows us to capture complex linguistic phenomena such as coreference, seman- tic roles, or polarity. When measuring the similarity between two AMR graphs A and B, for instance for the pur- pose of AMR parse quality evaluation, the metric of choice is usually SMATCH (Cai and Knight, 2013). Its backbone is an alignment-search be- tween the graphs’variables. Recently, the SEMBLEU metric (Song and Gildea, 2019) has been proposed that operates on the basis of a variable-free AMR (Figure 1, right),1 converting itto a bag of k-grams. Circumventing a variable alignment search redu- ces computational cost and ensures full determi- nacy.Also, grounding the metric in BLEU (Papineni et al., 2002) has a certain appeal, since BLEU is quite popular in machine translation. However, we find that we are lacking a principled in-depth comparison of the properties of different AMR metrics that would help informing researchers to answer questions such as: Which metric should I use to assess the similarity of two AMR graphs, e.g., in AMR parser evaluation? What are the trade-offs when choosing one metric over the other? Besides providing criteria for such a principled comparison, we discuss a property that none of the existing AMR metrics currently satisfies: They do not measure graded meaning differences.Such differences may emerge because of near-synonyms such as ruin – annihilate; skinny – thin – slim; enemy – foe (Inkpen and Hirst, 2006; Edmonds and Hirst, 2002) or paraphrases such as be able to – can; unclear – not clear. In a classical syntactic parsing task, metrics do not …

Content truncated to 500 words