Text Content and Task Performance in the Evaluation of a Natural Language Generation System

International Conference RANLP 2009 - Borovets, Bulgaria, pages 107–112 Text content and task performance in the evaluation of a Natural Language Generation system Albert Gatt Department of Computing Science University of Aberdeen a.gatt@abdn.ac.uk Franc¸ois Portet Laboratoire d’Informatique de Grenoble Grenoble Institute of Technology francois.portet@imag.fr Abstract An important question in the evaluation of Natu- ral Language Generation systems concerns the re- lationship between textual characteristics and task performance. If the results of task-based evalua- tion can be correlated to properties of the text, there are better prospects for improving the system. The present paper investigates this relationship by fo- cusing on the outcomes of a task-based evaluation of a system that generates summaries of patient data, attempting to correlate these with the results of an analysis of the system’s texts, compared to a set of gold standard human-authored summaries. Keywords Natural Language Generation, evaluation, decision support, domain on- tology 1 Introduction In the evaluation of NLP systems, an important distinction is that between intrinsic criteria, which typically measure aspects of output quality, and extrinsic ones, which assess a system in terms of its adequacy in its target setting [9]. The relationship between the two is important. A systematic relationship between the outcomes of extrinsic evaluation and properties of a system’s output can indicate directions for improvement in output, leading to improvements in the system’s utility in its target setting. This paper focusses on the evaluation of a a Natural Language Generation (NLG) system, BT-45 [13], which generates summaries of clinical patient data in a Neona- tal Intensive Care Unit (NICU). It was been evaluated in an experiment comparing decision making by clinicians based on the system output and other ways of presenting the same information, including human-authored summaries [17]. The aims of this paper are twofold. First, we investigate the relationship between intrinsic properties of generated text – particularly its informativeness and relevance – and its utility for decision-making. Second, we propose a novel intrinsic evaluation method. Rather than comparing BT-45 texts to a gold standard based on surface characteristics, such as matching n−grams, we make explicit use of do- main knowledge in the form of an ontology and attempt to quantify the differences in the content selection strategies underlying the BT-45 and the gold standard texts. 2 Background Intrinsic evaluation in NLG has often relied on human in- put, typically in the form of ratings of or responses to ques- tionnaires [12, 4, 7]. Automatic intrinsic methods exploit- ing corpora have mostly been used in evaluations of mor- phosyntactic realisers [11, 3]. Extrinsic, task-based meth- ods are also widespread [14, 10, 15]. While such studies tend to be more expensive and labour-intensive, extrinsic evaluation criteria give a reliable assessment of the sys- tem’s utility in doing what it was designed to do. The relationship between these different classes of evaluation methods is not straightforward. Recent work has shown that corpus-based intrinsic methods do not corre- late with the results of intrinsic evaluation based on human judgements, suggesting that they are …

Content truncated to 500 words