Towards a Decomposable Metric for Explainable Evaluation of Text Generation from AMR

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 1504–1518 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 1504 Towards a Decomposable Metric for Explainable Evaluation of Text Generation from AMR Juri Opitz Dept. of Computational Linguistics Heidelberg University 69120 Heidelberg opitz@cl.uni-heidelberg.de Anette Frank Dept. of Computational Linguistics Heidelberg University 69120 Heidelberg frank@cl.uni-heidelberg.de Abstract Systems that generate natural language text from abstract meaning representations such as AMR are typically evaluated using auto- matic surface matching metrics that compare the generated texts to reference texts from which the input meaning representations were constructed. We show that besides well- known issues from which such metrics suf- fer, an additional problem arises when apply- ing these metrics for AMR-to-text evaluation, since an abstract meaning representation al- lows for numerous surface realizations. In this work we aim to alleviate these issues by proposing MFβ, a decomposable metric that builds on two pillars. The first is the prin- ciple of meaning preservation M: it mea- sures to what extent a given AMR can be re- constructed from the generated sentence us- ing SOTA AMR parsers and applying (fine- grained) AMR evaluation metrics to measure the distance between the original and the re- constructed AMR. The second pillar builds on a principle of (grammatical) form F that measures the linguistic quality of the gener- ated text, which we implement using SOTA language models. In two extensive pilot stud- ies we show that fulfillment of both princi- ples offers benefits for AMR-to-text evalua- tion, including explainability of scores. Since MFβ does not necessarily rely on gold AMRs, it may extend to other text generation tasks. 1 Introduction Abstract Meaning Representation (AMR, Ba- narescu et al. (2013)) aims at capturing the meaning of a sentence in a machine-readable graph format. AMR captures, i.a., word senses, semantic roles and coreference. The AMR in Fig. 1 represents the sentence Perhaps, the parrot is telling itself a story. In this graph, tell-01 links to a PropBank (Palmer et al., 2005) frame, and argn labels indicate partici- possible-01 tell-01 parrot arg0 arg0 arg1 story arg2 Figure 1: “Perhaps, the parrot is telling itself a story”. pant roles: parrot is both speaker (arg0) and hearer (arg2), story is the utterance (arg1). The task of AMR-to-text generation has recently garnered much attention (Song et al., 2017, 2018; Konstas et al., 2017; Cai and Lam, 2020b; Ribeiro et al., 2019). The output of AMR-to-text systems is typically evaluated against the sentence from which the AMR was created, using standard surface string matching metrics such as BLEU (Papineni et al., 2002) or CHRF(++) (Stanojevi´c et al., 2015; Popovi´c, 2015, 2016; Popov, 2017), as is standard in many NLG tasks. These metrics suffer from sev- eral issues, for example, they penalize paraphrases, are highly sensitive to outliers (Mathur et al., 2020), and lack interpretability (Sai et al., 2020). Some of these issues get compounded when eval- uating AMR-to-text. The core of the problem is that there are many …

Content truncated to 500 words