Using a Randomised Controlled Clinical Trial to Evaluate an NLG System

Using a Randomised Controlled Clinical Trial to Evaluate an NLG System Ehud Reiter � Roma Robertson � A Scott Lennox � Liesl Osman � Departments of Computing Science � , General Practice � , and Medicine and Therapeutics � University of Aberdeen, Aberdeen, Scotland, UK � e.reiter, roma.robertson, s.lennox, l.osman � @abdn.ac.uk Abstract The STOP system, which generates personalised smoking-cessation letters, was evaluated by a randomised con- trolled clinical trial. We believe this is the largest and perhaps most rigorous task effectiveness evaluation ever per- formed on an NLG system. The de- tailed results of the clinical trial have been presented elsewhere, in the med- ical literature. In this paper we discuss the clinical trial itself: its structure and cost, what we did and did not learn from it (especially considering that the trial showed that STOP was not effective), and how it compares to other NLG eval- uation techniques. 1 Introduction There is increasing interest in techniques for eval- uating Natural Language Generation (NLG) sys- tems. However, we are not aware of any previ- ously reported evaluations of NLG systems which have rigorously compared the task effectiveness of an NLG system to a non-NLG alternative. In this paper we discuss such an evaluation, a large scale (2553 subjects) randomised controlled clin- ical trial which evaluated the effectiveness of per- sonalised smoking-cessation letters generated by the STOP system (Reiter et al., 1999). We be- lieve that this is the largest, most expensive, and perhaps most rigorous evaluation ever done of an NLG system; it was also a disappointing evalua- tion, as it showed that STOP letters in general were no more effective than control letters. The detailed results of the STOP evaluation have been presented elsewhere, in the medical lit- erature (Lennox et al., 2001). The purpose of this paper is to discuss the clinical trial from an NLG evaluation perspective, in order to help future re- searchers decide when a clinical trial (or similar large-scale task effectiveness evaluation) would be an appropriate way to evaluate their systems. 2 Evaluation of NLG Systems Evaluation is becoming increasingly important in NLG, as in other areas of NLP; see Mellish and Dale (1998) for a summary of NLG evaluation. As Mellish and Dale point out, we can evalu- ate the effectiveness of underlying theories, gen- eral properties of NLG systems and texts (such as computational speed, or text understandability), or the effectiveness of the generated texts in an actual task or application context. Theory eval- uations are typically done by comparing predic- tions of a theory to what is observed in a human- authored corpus (for example, (Yeh and Mellish, 1997)). Evaluations of text properties are typi- cally done by asking human judges to rate the quality of generated texts (for example, (Lester and Porter, 1997)); sometimes human-authored texts are included in the rated set (without judges knowing which texts are human-authored) to pro- vide a baseline. Task evaluations (for example, (Young, 1999)) are typically done by showing hu- man subjects different …

Content truncated to 500 words