Further Meta-Evaluation of Broad-Coverage Surface Realization

Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pages 564–574, MIT, Massachusetts, USA, 9-11 October 2010. c⃝2010 Association for Computational Linguistics Further Meta-Evaluation of Broad-Coverage Surface Realization Dominic Espinosa and Rajakrishnan Rajkumar and Michael White and Shoshana Berleant Department of Linguistics The Ohio State University Columbus, Ohio, USA {espinosa,raja,mwhite,berleant}@ling.ohio-state.edu Abstract We present the first evaluation of the utility of automatic evaluation metrics on surface real- izations of Penn Treebank data. Using outputs of the OpenCCG and XLE realizers, along with ranked WordNet synonym substitutions, we collected a corpus of generated surface re- alizations. These outputs were then rated and post-edited by human annotators. We eval- uated the realizations using seven automatic metrics, and analyzed correlations obtained between the human judgments and the auto- matic scores. In contrast to previous NLG meta-evaluations, we find that several of the metrics correlate moderately well with human judgments of both adequacy and fluency, with the TER family performing best overall. We also find that all of the metrics correctly pre- dict more than half of the significant system- level differences, though none are correct in all cases. We conclude with a discussion of the implications for the utility of such metrics in evaluating generation in the presence of varia- tion. A further result of our research is a cor- pus of post-edited realizations, which will be made available to the research community. 1 Introduction and Background In building surface-realization systems for natural language generation, there is a need for reliable automated metrics to evaluate the output. Unlike in parsing, where there is usually a single gold- standard parse for a sentence, in surface realization there are usually many grammatically-acceptable ways to express the same concept. This parallels the task of evaluating machine-translation (MT) sys- tems: for a given segment in the source language, there are usually several acceptable translations into the target language. As human evaluation of trans- lation quality is time-consuming and expensive, a number of automated metrics have been developed to evaluate the quality of MT outputs. In this study, we investigate whether the metrics developed for MT evaluation tasks can be used to reliably evaluate the outputs of surface realizers, and which of these metrics are best suited to this task. A number of surface realizers have been devel- oped using the Penn Treebank (PTB), and BLEU scores are often reported in the evaluations of these systems. But how useful is BLEU in this con- text? The original BLEU study (Papineni et al., 2001) scored MT outputs, which are of generally lower quality than grammar-based surface realiza- tions. Furthermore, even for MT systems, the usefulness of BLEU has been called into question (Callison-Burch et al., 2006). BLEU is designed to work with multiple reference sentences, but in tree- bank realization, there is only a single reference sen- tence available for comparison. A few other studies have investigated the use of such metrics in evaluating the output of NLG sys- tems, notably (Reiter and Belz, 2009) and (Stent …

Content truncated to 500 words