A Dynamic, Interpreted CheckList for Meaning-oriented NLG Metric Evaluation – through the Lens of Semantic Similarity Rating

Proceedings of the 11th Joint Conference on Lexical and Computational Semantics, pages 157 - 172 July 14-15, 2022 ©2022 Association for Computational Linguistics A Dynamic, Interpreted CheckList for Meaning-oriented NLG Metric Evaluation – through the Lens of Semantic Similarity Rating Laura Zeidler and Juri Opitz and Anette Frank Department of Computational Linguistics Heidelberg University, Germany {zeidler|opitz|frank}@cl.uni-heidelberg.de Abstract Evaluating the quality of generated text is diffi- cult, since traditional NLG evaluation metrics, focusing more on surface form than meaning, often fail to assign appropriate scores. This is especially problematic for AMR-to-text evalua- tion, given the abstract nature of AMR. Our work aims to support the development and improvement of NLG evaluation metrics that focus on meaning, by developing a dynamic CheckList for NLG metrics that is interpreted by being organized around meaning-relevant linguistic phenomena. Each test instance con- sists of a pair of sentences with their AMR graphs and a human-produced textual seman- tic similarity or relatedness score. Our Check- List facilitates comparative evaluation of met- rics and reveals strengths and weaknesses of novel and traditional metrics. We demonstrate the usefulness of CheckList by designing a new metric GRACO that computes lexical cohe- sion graphs over AMR concepts. Our analysis suggests that GRACO presents an interesting NLG metric worth future investigation and that meaning-oriented NLG metrics can profit from graph-based metric components using AMR. 1 Introduction Abstract Meaning Representation (AMR, Ba- narescu et al. (2013)) has become popular in NLP, one of the reasons being that AMR captures the essence of a sentence’s meaning, while abstract- ing away from syntactic idiosyncrasies. Especially AMR-to-text generation (Konstas et al., 2017; Song et al., 2018; Wang et al., 2020; Blloshmi et al., 2021) has received much attention for applications that require text generation from structured content. However, the evaluation of text generated from AMR has been argued to be unsatisfactory (Man- ning et al., 2020). Also, Opitz and Frank (2021) show that the syntactic diversity of sentences gener- ated from AMR is challenging for traditional NLG metrics, especially when candidates differ from the reference in surface properties. AMR AMR’ Snt Snt’ humScore phenomenon srcAMR candAMR candSnt goldSnt Presumed AMR2text generation amr_metrc(srcAMR, candAMR), a2t_metric(candSnt, srcAMR), nlg_metric(candSnt, goldSnt) ≈ humScore Data Setup Metric objective examples humScore phenomen. humScore phenomen. humScore phenomenon Figure 1: Our CheckList design for evaluating meaning- oriented NLG metrics against human semantic textual similarity and relatedness judgements – applicable to textual, meaning graph based and hybrid metrics. Several metrics have been proposed that aim to rate the similarity of the meaning of sentences or phrases (Zhang et al. (2020); Opitz and Frank (2021); Zhao et al. (2019)). However, it is difficult to judge where exactly such a metric fails, mak- ing it hard for developers to further improve it. To address similar problems, Ribeiro et al. (2020) re- cently proposed a "task-agnostic methodology for testing NLP models" called CheckList. They ar- gue that such a method should be used for testing NLP systems instead of solely relying on automatic metrics, which can overestimate …

Content truncated to 500 words