RoMe: A Robust Metric for Evaluating Natural Language Generation

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 5645 - 5657 May 22-27, 2022 c⃝2022 Association for Computational Linguistics RoMe: A Robust Metric for Evaluating Natural Language Generation Md Rashad Al Hasan Rony1,3, Liubov Kovriguina3, Debanjan Chaudhuri1, Ricardo Usbeck2, Jens Lehmann1,3 1University of Bonn, 2University of Hamburg, 3Fraunhofer IAIS Dresden {rashad.rony,liubov.kovriguina,jens.lehmann}@iais.fraunhofer.de {lehmann,d.chaudhuri}@uni-bonn.de ricardo.usbeck@uni-hamburg.de Abstract Evaluating Natural Language Generation (NLG) systems is a challenging task. Firstly, the metric should ensure that the generated hypothesis reflects the reference’s semantics. Secondly, it should consider the grammatical quality of the generated sentence. Thirdly, it should be robust enough to handle various surface forms of the generated sentence. Thus, an effective evaluation metric has to be multifaceted. In this paper, we propose an automatic evaluation metric incorporating several core aspects of natural language un- derstanding (language competence, syntactic and semantic variation). Our proposed metric, RoMe, is trained on language features such as semantic similarity combined with tree edit distance and grammatical acceptability, using a self-supervised neural network to assess the overall quality of the generated sentence. Moreover, we perform an extensive robustness analysis of the state-of-the-art methods and RoMe. Empirical results suggest that RoMe has a stronger correlation to human judgment over state-of-the-art metrics in evaluating system-generated sentences across several NLG tasks. 1 Introduction Automatic generation of fluent and coherent nat- ural language is a key step for human-computer interaction. Evaluating generative systems such as text summarization, dialogue systems, and machine translation is challenging since the assessment in- volves several criteria such as content determina- tion, lexicalization, and surface realization (Liu et al., 2016; Dale and Mellish, 1998). For assess- ing system-generated outputs, human judgment is considered to be the best approach. Obtaining hu- man evaluation ratings, on the other hand, is both expensive and time-consuming. As a result, devel- oping automated metrics for assessing the quality of machine-generated text has become an active area of research in NLP. The quality estimation task primarily entails determining the similarity between the reference and hypothesis as well as assessing the hypoth- esis for grammatical correctness and naturalness. Widely used evaluation metrics such as BLEU (Pap- ineni et al., 2002), METEOR (Banerjee and Lavie, 2005), and ROUGE (Lin, 2004) which compute the word-overlaps, were primarily designed for evaluating machine translation and text summa- rization systems. Word-overlap based metrics, on the other hand, are incapable of capturing the hy- potheses’ naturalness and fluency. Furthermore, they do not consider the syntactic difference be- tween reference and hypothesis. In a different line of research, word mover distance (WMD) (Kus- ner et al., 2015), BERTScore (Zhang et al., 2020a) and MoverScore (Zhao et al., 2019) compute word embedding based similarity for evaluating system- generated texts. Although these metrics employ the contextualized representation of words, they do not take the grammatical acceptability of the hypoth- esis and the syntactical similarity to the reference into account. To address these shortcomings, we propose RoMe, an automatic and robust metric for eval- uating NLG systems. RoMe employs …

Content truncated to 500 words