Towards the Evaluation of Referring Expression Generation

Proceedings of the 2006 Australasian Language Technology Workshop (ALTW2006), pages 115–122. Towards the Evaluation of Referring Expression Generation Jette Viethen Centre for Language Technology Macquarie University Sydney NSW 2109 jviethen@ics.mq.edu.au Robert Dale Centre for Language Technology Macquarie University Sydney NSW 2109 robert.dale@mq.edu.au Abstract The Natural Language Generation com- munity is currently engaged in discussion as to whether and how to introduce one or several shared evaluation tasks, as are found in other fields of Natural Language Processing. As one of the most well- defined subtasks in NLG, the generation of referring expressions looks like a strong candidate for piloting such shared tasks. Based on our earlier evaluation of a num- ber of existing algorithms for the genera- tion of referring expressions, we explore in this paper some problems that arise in designing an evaluation task in this field, and try to identify general considerations that need to be met in evaluating genera- tion subtasks. 1 Introduction In recent years, the inclusion of an evaluation com- ponent has become almost obligatory in any pub- lication in the field of Natural Language Process- ing. For complete systems, user-based and task- oriented evaluation are almost standard practice in both the Natural Language Understanding (NLU) and Natural Language Generation (NLG) commu- nities. A third, more competitive, form of eval- uation has become increasingly popular in NLU in the form of shared-task evaluation campaigns (STECs). In a STEC, different approaches to a well- defined problem are compared based on their per- formance on the same task. A large number of dif- ferent research communities within NLP, such as Question Answering, Machine Translation, Doc- ument Summarisation, Word Sense Disambigua- tion, and Information Retrieval, have adopted a shared evaluation metric and in many cases a shared-task evaluation competition. The NLG community has so far withstood this trend towards a joint evaluation metric and a com- petitive evaluation task, but the idea has surfaced in a number of discussions, and most intensely at the 2006 International Natural Language Genera- tion Conference (see, for example, Bangalore et al. (2000), Reiter and Sripada (2002), Reiter and Belz (2006), Belz and Reiter (2006), Belz and Kilgarriff (2006), Paris et al. (2006), and van Deemter et al. (2006)). Amongst the various component tasks that make up Natural Language Generation, the gen- eration of referring expressions is probably the subtask for which there is the most agreement on problem definition; a significant body of work now exists in the development of algorithms for gener- ating referring expressions, with almost all pub- lished contributions agreeing on the general char- acterisation of the task and what constitutes a so- lution. This suggests that, if formal shared tasks for NLG are to be developed, the generation of re- ferring expressions is a very strong candidate. In (Viethen and Dale, 2006), we argued that the evaluation of referring expression generation algo- rithms against natural, human-generated data is of fundamental importance in assessing their useful- ness for the generation of understandable, natural- sounding referring expressions. In this paper, …

Content truncated to 500 words