Human Evaluation of a German Surface Realisation Ranker

Proceedings of the 12th Conference of the European Chapter of the ACL, pages 112–120, Athens, Greece, 30 March – 3 April 2009. c⃝2009 Association for Computational Linguistics Human Evaluation of a German Surface Realisation Ranker Aoife Cahill Institut f¨ur Maschinelle Sprachverarbeitung (IMS) University of Stuttgart 70174 Stuttgart, Germany aoife.cahill@ims.uni-stuttgart.de Martin Forst Palo Alto Research Center 3333 Coyote Hill Road Palo Alto, CA 94304, USA mforst@parc.com Abstract In this paper we present a human-based evaluation of surface realisation alterna- tives. We examine the relative rankings of naturally occurring corpus sentences and automatically generated strings chosen by statistical models (language model, log- linear model), as well as the naturalness of the strings chosen by the log-linear model. We also investigate to what extent preced- ing context has an effect on choice. We show that native speakers do accept quite some variation in word order, but there are also clearly factors that make certain real- isation alternatives more natural. 1 Introduction An important component of research on surface realisation (the task of generating strings for a given abstract representation) is evaluation, espe- cially if we want to be able to compare across sys- tems. There is consensus that exact match with respect to an actually observed corpus sentence is too strict a metric and that BLEU score measured against corpus sentences can only give a rough im- pression of the quality of the system output. It is unclear, however, what kind of metric would be most suitable for the evaluation of string realisa- tions, so that, as a result, there have been a range of automatic metrics applied including inter alia ex- act match, string edit distance, NIST SSA, BLEU, NIST, ROUGE, generation string accuracy, gener- ation tree accuracy, word accuracy (Bangalore et al., 2000; Callaway, 2003; Nakanishi et al., 2005; Velldal and Oepen, 2006; Belz and Reiter, 2006). It is not always clear how appropriate these met- rics are, especially at the level of individual sen- tences. Using automatic evaluation metrics cannot be avoided, but ideally, a metric for the evaluation of realisation rankers would rank alternative real- isations in the same way as native speakers of the language for which the surface realisation system is developed, and not only globally, but also at the level of individual sentences. Another major consideration in evaluation is what to take as the gold standard. The easiest op- tion is to take the original corpus string that was used to produce the abstract representation from which we generate. However, there may well be other realisations of the same input that are as suitable in the given context. Reiter and Sripada (2002) argue that while we should take advantage of large corpora in NLG, we also need to take care that we do not introduce errors by learning from incorrect data present in corpora. In order to better understand what makes good evaluation data (and metrics), we designed and im- plemented an experiment in which human judges evaluated German string realisations. The main aims of this …

Content truncated to 500 words