Building an Evaluation Scale using Item Response Theory

Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 648–657, Austin, Texas, November 1-5, 2016. c⃝2016 Association for Computational Linguistics Building an Evaluation Scale using Item Response Theory John P. Lalor1, Hao Wu2, Hong Yu1,3 1 University of Massachusetts, MA, USA 2 Boston College, MA, USA 3 Bedford VAMC and CHOIR, MA, USA lalor@cs.umass.edu, hao.wu.5@bc.edu, hong.yu@umassmed.edu Abstract Evaluation of NLP methods requires test- ing against a previously vetted gold-standard test set and reporting standard metrics (ac- curacy/precision/recall/F1). The current as- sumption is that all items in a given test set are equal with regards to difficulty and discrim- inating power. We propose Item Response Theory (IRT) from psychometrics as an alter- native means for gold-standard test-set gener- ation and NLP system evaluation. IRT is able to describe characteristics of individual items - their difficulty and discriminating power - and can account for these characteristics in its es- timation of human intelligence or ability for an NLP task. In this paper, we demonstrate IRT by generating a gold-standard test set for Recognizing Textual Entailment. By collect- ing a large number of human responses and fitting our IRT model, we show that our IRT model compares NLP systems with the per- formance in a human population and is able to provide more insight into system performance than standard evaluation metrics. We show that a high accuracy score does not always im- ply a high IRT score, which depends on the item characteristics and the response pattern.1 1 Introduction Advances in artificial intelligence have made it pos- sible to compare computer performance directly with human intelligence (Campbell et al., 2002; Fer- rucci et al., 2010; Silver et al., 2016). In most cases, a common approach to evaluating the performance 1Data and code will be made available for download at https://people.cs.umass.edu/lalor/irt.html of a new system is to compare it against an unseen gold-standard test dataset (GS items). Accuracy, re- call, precision and F1 scores are commonly used to evaluate NLP applications. These metrics assume that GS items have equal weight for evaluating per- formance. However, individual items are different: some may be so hard that most/all NLP systems an- swer incorrectly; others may be so easy that every NLP system answers correctly. Neither item type provides meaningful information about the perfor- mance of an NLP system. Items that are answered incorrectly by some systems and correctly by oth- ers are useful for differentiating systems according to their individual characteristics. In this paper we introduce Item Response The- ory (IRT) from psychometrics and demonstrate its application to evaluating NLP systems. IRT is a the- ory of evaluation for characterizing test items and estimating human ability from their performance on such tests. IRT assumes that individual test ques- tions (referred to as “items” in IRT) have unique characteristics such as difficulty and discriminating power. These characteristics can be identified by fit- ting a joint model of human ability and item charac- teristics to human response patterns to the test items. Items that …

Content truncated to 500 words