Automatically Evaluating Answers to Definition Questions

Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing (HLT/EMNLP), pages 931–938, Vancouver, October 2005. c⃝2005 Association for Computational Linguistics Automatically Evaluating Answers to Definition Questions Jimmy Lin1,3 and Dina Demner-Fushman2,3 1College of Information Studies 2Department of Computer Science 3Institute for Advanced Computer Studies University of Maryland College Park, MD 20742, USA jimmylin@umd.edu, demner@cs.umd.edu Abstract Following recent developments in the au- tomatic evaluation of machine translation and document summarization, we present a similar approach, implemented in a mea- sure called POURPRE, for automatically evaluating answers to definition questions. Until now, the only way to assess the cor- rectness of answers to such questions in- volves manual determination of whether an information nugget appears in a sys- tem’s response. The lack of automatic methods for scoring system output is an impediment to progress in the field, which we address with this work. Experiments with the TREC 2003 and TREC 2004 QA tracks indicate that rankings produced by our metric correlate highly with official rankings, and that POURPRE outperforms direct application of existing metrics. 1 Introduction Recent interest in question answering has shifted away from factoid questions such as “What city is the home to the Rock and Roll Hall of Fame?”, which can typically be answered by a short noun phrase, to more complex and difficult questions. One interesting class of information needs con- cerns so-called definition questions such as “Who is Vlad the Impaler?”, whose answers would include “nuggets” of information about the 16th century warrior prince’s life, accomplishments, and legacy. Actually a misnomer, definition questions can be better paraphrased as “Tell me interesting things about X.”, where X can be a person, an organiza- tion, a common noun, etc. Taken another way, defi- nition questions might be viewed as simultaneously asking a whole series of factoid questions about the same entity (e.g., “When was he born?”, “What was his occupation?”, “Where did he live?”, etc.), except that these questions are not known in advance; see Prager et al. (2004) for an implementation based on this view of definition questions. Much progress in natural language processing and information retrieval has been driven by the creation of reusable test collections. A test collection con- sists of a corpus, a series of well-defined tasks, and a set of judgments indicating the “correct answers”. To complete the picture, there must exist meaning- ful metrics to evaluate progress, and ideally, a ma- chine should be able to compute these values auto- matically. Although “answers” to definition ques- tions are known, there is no way to automatically and objectively determine if they are present in a given system’s response (we will discuss why in Section 2). The experimental cycle is thus tortuously long; to accurately assess the performance of new techniques, one must essentially wait for expensive, large-scale evaluations that employ human assessors to judge the runs (e.g., the TREC QA track). This situation mirrors the state of machine translation and document summarization research a few years ago. Since …

Content truncated to 500 words