A Probabilistic Rasch Analysis of Question Answering Evaluations

A Probabilistic Rasch Analysis of Question Answering Evaluations Rense Lange Integrated Knowledge Systems rlange@iknowsys.org Warren R. Greiff The MITRE Corporation greiff@mitre.org Juan Moran University of Illinois, Urbana-Champaign jmoran@ncsa.uiuc.edu Lisa Ferro The MITRE Corporation lferro@mitre.org Abstract The field of Psychometrics routinely grapples with the question of what it means to measure the inherent ability of an organism to perform a given task, and for the last forty years, the field has increasingly relied on probabilistic methods such as the Rasch model for test con- struction and the analysis of test results. Be- cause the underlying issues of measuring ability apply to human language technologies as well, such probabilistic methods can be ad- vantageously applied to the evaluation of those technologies. To test this claim, Rasch measurement was applied to the results of 67 systems participating in the Question Answer- ing track of the 2002 Text REtrieval Confer- ence (TREC) competition. Satisfactory model fit was obtained, and the paper illustrates the theoretical and practical strengths of Rasch scaling for evaluating systems as well as ques- tions. Most important, simulations indicate that a test invariant metric can be defined by carrying forward 20 to 50 equating questions, thus placing the yearly results on a common scale. 1 Introduction For a number of years, objective evaluation of state-of- the-art computational systems on realistic language processing tasks has been a driving force in the advance of Human Language Technology (HLT). Often, such evaluations are based on the use of simple sum-scores (i.e., the number of correct answers) and derivatives thereof (e.g., percentages), or on ad-hoc ways to rank or order system responses according to their correctness. Unfortunately, research in other areas indicates that such approaches rarely yield a cumulative body of knowledge, thereby complicating theory formation and practical decision making alike. In fact, although it is often taken for granted that sums or percentages ade- quately reflect systems’ performance, this assumption does not agree with many models currently used in edu- cational testing (cf., Hambleton and Swaminathan, 1985; Stout, 2002). To address this situation, we present the use of Rasch (1960/1980) measurement to the HLT research community, in general, and to the Question Answering (QA) research community, in particular. Rasch measurement has evolved over the last forty years to rigorously quantify performance aspects in such diverse areas as educational testing, cognitive develop- ment, moral judgment, eating disorders (see e.g., Bond and Fox, 2001), as well as olfactory screening for Alz- heimer’s disease (Lange et al., 2002) and model glider competitions (Lange, 2003). In each case, the major contribution of Rasch measurement is to decompose performance into two additive sources: the difficulty of the task and the ability of the person or system perform- ing this task. While Rasch measurement is new to the evaluation of the performance of HLT systems, we in- tend to demonstrate that this approach applies here as well, and that it potentially provides significant advan- tages over traditional evaluation approaches. Our principal theoretical argument in favor of Rasch modeling is that …

Content truncated to 500 words