Curious Case of Language Generation Evaluation Metrics: A Cautionary Tale

Proceedings of the 28th International Conference on Computational Linguistics, pages 2322–2328 Barcelona, Spain (Online), December 8-13, 2020 2322 Curious Case of Language Generation Evaluation Metrics: A Cautionary Tale Ozan Caglayan1, Pranava Madhyastha1 and Lucia Specia1,2,3 Imperial College London1, University of Sheffield2, ADAPT - Dublin City University3 o.caglayan@ic.ac.uk, pranava@ic.ac.uk, l.specia@ic.ac.uk Abstract Automatic evaluation of language generation systems is a well-studied problem in Natural Lan- guage Processing. While novel metrics are proposed every year, a few popular metrics remain as the de facto metrics to evaluate tasks such as image captioning and machine translation, de- spite their known limitations. This is partly due to ease of use, and partly because researchers expect to see them and know how to interpret them. In this paper, we urge the community for more careful consideration of how they automatically evaluate their models by demonstrating important failure cases on multiple datasets, language pairs and tasks. Our experiments show that metrics (i) usually prefer system outputs to human-authored texts, (ii) can be insensitive to correct translations of rare words, (iii) can yield surprisingly high scores when given a single sentence as system output for the entire test set. 1 Introduction Human assessment is the best practice at hand for evaluating language generation tasks such as machine translation (MT), dialogue systems, visual captioning and abstractive summarisation. In practice, how- ever, we rely on automatic metrics which compare system outputs to human-authored references. Initially proposed for MT evaluation, metrics such as BLEU (Papineni et al., 2002) and METEOR (Denkowski and Lavie, 2014) are increasingly used for other tasks, along with more task-oriented metrics such as ROUGE (Lin, 2004) for summarisation and CIDER (Vedantam et al., 2015) for visual captioning. Reiter and Belz (2009) remark that it is not sufficient to conclude on the usefulness of a natural lan- guage generation system’s output by solely relying on metrics that quantify the similarity of the output to human-authored texts. Previous criticisms concerning automatic metrics corroborate this perspective to some degree. To cite a few, in the context of MT, Callison-Burch et al. (2006) state that human judgments may not correlate with BLEU and an increase in BLEU does not always indicate an improvement in qual- ity. Further, Mathur et al. (2020) challenge the stability of the common practices measuring correlation between metrics and human judgments, the standard approach in the MT community, and show that they may be severely impacted by outlier systems and the sample size. In the context of image captioning, Wang and Chan (2019) claim that the consensus-based evaluation protocol of CIDER actually penalises output diversity. Similar problems have also been discussed in the area of automatic summarisation with respect to ROUGE (Schluter, 2017). Nevertheless, automatic metrics like these are a necessity and remain popular, especially given the increasing number of open evaluation challenges. In this paper, we further probe BLEU, METEOR, CIDER-D and ROUGEL metrics (§ 2) that are com- monly used to quantify progress in language generation tasks. We also probe a recently proposed contex- tualised …

Content truncated to 500 words