Perturbation CheckLists for Evaluating NLG Evaluation Metrics

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7219–7234 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 7219 Perturbation CheckLists for Evaluating NLG Evaluation Metrics Ananya B. Sai1,2 Tanay Dixit1 Dev Yashpal Sheth1,2 Sreyas Mohan3 Mitesh M. Khapra1,2,4 1 Indian Institute of Technology, Madras 2 Robert Bosch Centre for Data Science and Artificial Intelligence, IIT Madras 3 Center for Data Science, New York University 4 AI4Bharat {ananya,devsheth,miteshk}@cse.iitm.ac.in, dixittanay@gmail.com , sm7582@nyu.edu Abstract Natural Language Generation (NLG) evalua- tion is a multifaceted task requiring assess- ment of multiple desirable criteria, e.g., flu- ency, coherency, coverage, relevance, ade- quacy, overall quality, etc. Across existing datasets for 6 NLG tasks, we observe that the human evaluation scores on these multiple cri- teria are often not correlated. For example, there is a very low correlation between human scores on fluency and data coverage for the task of structured data to text generation. This suggests that the current recipe of proposing new automatic evaluation metrics for NLG by showing that they correlate well with scores as- signed by humans for a single criteria (over- all quality) alone is inadequate. Indeed, our extensive study involving 25 automatic evalu- ation metrics across 6 different tasks and 18 different evaluation criteria shows that there is no single metric which correlates well with hu- man scores on all desirable criteria, for most NLG tasks. Given this situation, we propose CheckLists for better design and evaluation of automatic metrics. We design templates which target a specific criteria (e.g., coverage) and perturb the output such that the quality gets affected only along this specific criteria (e.g., the coverage drops). We show that existing evaluation metrics are not robust against even such simple perturbations and disagree with scores assigned by humans to the perturbed output. The proposed templates thus allow for a fine-grained assessment of automatic evalua- tion metrics exposing their limitations and will facilitate better design, analysis and evaluation of such metrics.1 1 Introduction As the number of tasks and benchmarks for NLG have increased (Gehrmann et al., 2021), the chal- lenges in evaluating NLG systems have also con- tinued to grow (Liu et al., 2016; Nema and Khapra, 1Our templates and code are available at https://iitmnlp.github.io/EvalEval/ 2018; Sai et al., 2019). One reliable way of evalu- ating NLG systems is to collect human judgements. However, this is a time consuming and expensive process (Freitag et al., 2021; Deriu et al., 2019; Howcroft et al., 2020). Hence, automatic evalua- tion metrics such as BLEU (Papineni et al., 2002) which are quicker to compute have become popular, despite being less reliable (Callison-Burch et al., 2006; Reiter, 2018). The survey by Sai et al. (2020b) shows that more than 35 automatic evaluation metrics have been proposed for NLG since 2014, however, there is no careful evaluation of the ability of such metrics to assess the quality of the output of an NLG system on multiple desired criteria. For example, con- sider the task of dialog evaluation, where humans are …

Content truncated to 500 words