MUTT: Metric Unit TesTing for Language Generation Tasks

Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 1935–1943, Berlin, Germany, August 7-12, 2016. c⃝2016 Association for Computational Linguistics MUTT: Metric Unit TesTing for Language Generation Tasks Willie Boag, Renan Campos, Kate Saenko, Anna Rumshisky Dept. of Computer Science University of Massachusetts Lowell 198 Riverside St, Lowell, MA 01854 {wboag,rcampos,saenko,arum}@cs.uml.edu Abstract Precise evaluation metrics are important for assessing progress in high-level lan- guage generation tasks such as machine translation or image captioning. Histor- ically, these metrics have been evaluated using correlation with human judgment. However, human-derived scores are often alarmingly inconsistent and are also limited in their ability to identify precise areas of weakness. In this paper, we perform a case study for metric evaluation by measuring the effect that systematic sentence trans- formations (e.g. active to passive voice) have on the automatic metric scores. These sentence “corruptions” serve as unit tests for precisely measuring the strengths and weaknesses of a given metric. We find that not only are human annotations heavily in- consistent in this study, but that the Met- ric Unit TesT analysis is able to capture precise shortcomings of particular metrics (e.g. comparing passive and active sen- tences) better than a simple correlation with human judgment can. 1 Introduction The success of high-level language generation tasks such as machine translation (MT), paraphrasing and image/video captioning depends on the existence of reliable and precise automatic evaluation metrics. Figure 1: A few select entries from the SICK dataset. All of these entries follow the same “Negated Subject” transfor- mation between sentence 1 and sentence 2, yet humans anno- tated them with an inconsistently wide range of scores (from 1 to 5). Regardless of whether the gold labels for this partic- ular transformation should score this high or low, they should score be scored consistently. Efforts have been made to create standard met- rics (Papineni et al., 2001; Lin, 2004; Denkowski and Lavie, 2014; Vedantam et al., 2014) to help advance the state-of-the-art. However, most such popular metrics, despite their wide use, have seri- ous deficiencies. Many rely on ngram matching and assume that annotators generate all reasonable refer- ence sentences, which is infeasible for many tasks. Furthermore, metrics designed for one task, e.g., MT, can be a poor fit for other tasks, e.g., video cap- tioning. To design better metrics, we need a principled approach to evaluating their performance. Histori- cally, MT metrics have been evaluated by how well they correlate with human annotations (Callison- Burch et al., 2010; Machacek and Bojar, 2014). However, as we demonstrate in Sec. 5, human judgment can result in inconsistent scoring. This presents a serious problem for determining whether 1935 a metric is ”good” based on correlation with incon- sistent human scores. When ”gold” target data is unreliable, even good metrics can appear to be inac- curate. Furthermore, correlation of system output with human-derived scores typically provides an overall score but fails to isolate specific errors that met- rics tend to miss. This makes it difficult to dis- …

Content truncated to 500 words