Will Pyramids Built of Nuggets Topple Over?

Proceedings of the Human Language Technology Conference of the North American Chapter of the ACL, pages 383–390, New York, June 2006. c⃝2006 Association for Computational Linguistics Will Pyramids Built of Nuggets Topple Over? Jimmy Lin1,2,3 and Dina Demner-Fushman2,3 1College of Information Studies 2Department of Computer Science 3Institute for Advanced Computer Studies University of Maryland College Park, MD 20742, USA jimmylin@umd.edu, demner@cs.umd.edu Abstract The present methodology for evaluating complex questions at TREC analyzes an- swers in terms of facts called “nuggets”. The official F-score metric represents the harmonic mean between recall and pre- cision at the nugget level. There is an implicit assumption that some facts are more important than others, which is im- plemented in a binary split between “vi- tal” and “okay” nuggets. This distinc- tion holds important implications for the TREC scoring model—essentially, sys- tems only receive credit for retrieving vi- tal nuggets—and is a source of evalua- tion instability. The upshot is that for many questions in the TREC testsets, the median score across all submitted runs is zero. In this work, we introduce a scor- ing model based on judgments from mul- tiple assessors that captures a more refined notion of nugget importance. We demon- strate on TREC 2003, 2004, and 2005 data that our “nugget pyramids” address many shortcomings of the present methodology, while introducing only minimal additional overhead on the evaluation flow. 1 Introduction The field of question answering has been moving away from simple “factoid” questions such as “Who invented the paper clip?” to more complex informa- tion needs such as “Who is Aaron Copland?” and “How have South American drug cartels been using banks in Liechtenstein to launder money?”, which cannot be answered by simple named-entities. Over the past few years, NIST through the TREC QA tracks has implemented an evaluation methodology based on the notion of “information nuggets” to as- sess the quality of answers to such complex ques- tions. This paradigm has gained widespread accep- tance in the research community, and is currently be- ing applied to evaluate answers to so-called “defini- tion”, “relationship”, and “opinion” questions. Since quantitative evaluation is arguably the sin- gle biggest driver of advances in language technolo- gies, it is important to closely examine the charac- teristics of a scoring model to ensure its fairness, re- liability, and stability. In this work, we identify a potential source of instability in the nugget evalua- tion paradigm, develop a new scoring method, and demonstrate that our new model addresses some of the shortcomings of the original method. It is our hope that this more-refined evaluation model can better guide the development of technology for an- swering complex questions. This paper is organized as follows: Section 2 provides a brief overview of the nugget evaluation methodology. Section 3 draws attention to the vi- tal/okay nugget distinction and the problems it cre- ates. Section 4 outlines our proposal for building “nugget pyramids”, a more-refined model of nugget importance that combines judgments from multiple assessors. Section 5 describes the methodology …

Content truncated to 500 words