Evaluation in Natural Language Generation: Lessons from Referring Expression Generation

Evaluation in Natural Language Generation: Lessons from Referring Expression Generation Jette Viethen — Robert Dale Centre for Language Technology Division of Information and Communication Sciences Macquarie University Sydney NSW 2109 Australia jviethen@ics.mq.edu.au robert.dale@mq.edu.au ABSTRACT. As one of the most well-defined subtasks in Natural Language Generation (NLG), the generation of referring expressions looks like a strong candidate for piloting shared evaluation tasks. Different to other areas of Natural Language Processing, it is still unclear what benefit the introduction of such tasks might have for the field of NLG. Based on an earlier evaluation of a number of well-established algorithms for the generation of referring expressions, this paper explores several problems that arise in designing evaluation for this task, and identifies general considerations that need to be met in evaluating Natural Language Generation subtasks. RÉSUMÉ. La génération d’expressions référentielles, une des sous-tâche de la génération auto- matique de textes les mieux définies, apparaît comme une candidate sérieuse pour la mise en place de tâches d’évaluation partagée, dans un domaine du traitement automatique des langues où la question de l’intérêt de ces tâches reste ouverte. Sur la base des résultats d’une évalua- tion de certains des principaux algorithmes connus de génération d’expressions référentielles, cet article explore plusieurs problèmes posés par l’évaluation et présente quelques considéra- tions d’ordre général à prendre en compte lors de l’évaluation des sous-tâches de la génération automatique de textes. KEYWORDS: Referring Expression Generation, Natural Language Generation, Evaluation MOTS-CLÉS : génération d’expressions référentielles, génération automatique de textes, évalua- tion TAL. Volume 48 – n˚1/2007, pages 141 à 160 142 TAL. Volume 48 – n˚1/2007 1. Introduction How do we know that something is working well, or as it is supposed to? Whether we call the relevant activity ‘testing’, ‘review’ or ‘assessment’, the fundamental task to be carried out is one of evaluation. We measure the behaviour of the artefact against some commonly accepted benchmark, or using some commonly accepted metric. Over the last 15 years, evaluation has taken on a key role in furthering research in the field of Natural Language Processing (NLP): this testing is generally carried out by embodying theoretical hypotheses in the form of implemented software, and then evaluating the behaviour of these implemented systems against collections of test data. For many of the component subtasks that make up Natural Language Understanding (NLU), and for many standard applications of NLP, this practice has become almost in- stitutionalised in the form of shared task evaluation campaigns (STECs). In a STEC, multiple teams will produce systems that attempt to solve a particular problem (the shared task); the systems will be developed on the basis of a set of training data, and then each is evaluated against the same previously unseen collection of test data, en- couraging a degree of competition. A large number of different research communities within NLP, such as Question Answering, Machine Translation, Document Summari- sation, Word Sense Disambiguation, and Information Retrieval, have adopted a shared evaluation metric and a shared-task evaluation competition. However, not all subfields …

Content truncated to 500 words