Different Structures for Evaluating Answers to Complex Questions: Pyramids Won’t Topple, and Neither Will Human Assessors

Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pages 768–775, Prague, Czech Republic, June 2007. c⃝2007 Association for Computational Linguistics Different Structures for Evaluating Answers to Complex Questions: Pyramids Won’t Topple, and Neither Will Human Assessors Hoa Trang Dang Information Access Division National Institute of Standards and Technology Gaithersburg, MD 20899 hoa.dang@nist.gov Jimmy Lin College of Information Studies University of Maryland College Park, MD 20742 jimmylin@umd.edu Abstract The idea of “nugget pyramids” has re- cently been introduced as a refinement to the nugget-based methodology used to evaluate answers to complex questions in the TREC QA tracks. This paper examines data from the 2006 evaluation, the first large-scale de- ployment of the nugget pyramids scheme. We show that this method of combining judgments of nugget importance from multi- ple assessors increases the stability and dis- criminative power of the evaluation while in- troducing only a small additional burden in terms of manual assessment. We also con- sider an alternative method for combining assessor opinions, which yields a distinction similar to micro- and macro-averaging in the context of classification tasks. While the two approaches differ in terms of underly- ing assumptions, their results are neverthe- less highly correlated. 1 Introduction The emergence of question answering (QA) systems for addressing complex information needs has ne- cessitated the development and refinement of new methodologies for evaluating and comparing sys- tems. In the Text REtrieval Conference (TREC) QA tracks organized by the U.S. National Institute of Standards and Technology (NIST), improvements in evaluation processes have kept pace with the evolu- tion of QA tasks. For the past several years, NIST has implemented an evaluation methodology based on the notion of “information nuggets” to assess an- swers to complex questions. As it has become the de facto standard for evaluating such systems, the research community stands to benefit from a better understanding of the characteristics of this evalua- tion methodology. This paper explores recent refinements to the nugget-based evaluation methodology developed by NIST. In particular, we examine the recent so-called “pyramid extension” that incorporates relevance judgments from multiple assessors to improve eval- uation stability (Lin and Demner-Fushman, 2006). We organize our discussion as follows: The next section begins by providing a brief overview of nugget-based evaluations and the pyramid exten- sion. Section 3 presents results from the first large- scale implementation of nugget pyramids for QA evaluation in TREC 2006. Analysis shows that this extension improves both stability and discriminative power. In Section 4, we discuss an alternative for combining multiple judgments that parallels the dis- tinction between micro- and macro-averaging often seen in classification tasks. Experiments reveal that the methods yield almost exactly the same results, despite operating on different granularities (individ- ual nuggets vs. individual users). 2 Evaluating Complex Questions Complex questions are distinguished from factoid questions such as “Who shot Abraham Lincoln?” in that they cannot be answered by named entities (e.g., persons, organizations, dates, etc.). Typically, these information needs are embedded in the context of a scenario (i.e., …

Content truncated to 500 words