A Simple Measure to Assess Non-response

Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics, pages 1415–1424, Portland, Oregon, June 19-24, 2011. c⃝2011 Association for Computational Linguistics A Simple Measure to Assess Non-response Anselmo Pe˜nas and Alvaro Rodrigo UNED NLP & IR Group Juan del Rosal, 16 28040 Madrid, Spain {anselmo,alvarory@lsi.uned.es} Abstract There are several tasks where is preferable not responding than responding incorrectly. This idea is not new, but despite several previous at- tempts there isn’t a commonly accepted mea- sure to assess non-response. We study here an extension of accuracy measure with this fea- ture and a very easy to understand interpreta- tion. The measure proposed (c@1) has a good balance of discrimination power, stability and sensitivity properties. We show also how this measure is able to reward systems that main- tain the same number of correct answers and at the same time decrease the number of in- correct ones, by leaving some questions unan- swered. This measure is well suited for tasks such as Reading Comprehension tests, where multiple choices per question are given, but only one is correct. 1 Introduction There is some tendency to consider that an incorrect result is simply the absence of a correct one. This is particularly true in the evaluation of Information Re- trieval systems where, in fact, the absence of results sometimes is the worse output. However, there are scenarios where we should consider the possibility of not responding, because this behavior has more value than responding incor- rectly. For example, during the process of introduc- ing new features in a search engine it is important to preserve users’ confidence in the system. Thus, a system must decide whether it should give or not a result in the new fashion or keep on with the old kind of output. A similar example is the decision about showing or not ads related to the query. Show- ing wrong ads harms the business model more than showing nothing. A third example more related to Natural Language Processing is the Machine Read- ing evaluation through reading comprehension tests. In this case, where multiple choices for a question are offered, choosing a wrong option should be pun- ished against leaving the question unanswered. In the latter case, the use of utility functions is a very common option. However, utility functions give arbitrary value to not responding and ignore the system’s behavior showed when it responds (see Section 2). To avoid this, we present c@1 measure (Section 2.2), as an extension of accuracy (the pro- portion of correctly answered questions). In Sec- tion 3 we show that no other extension produces a sensible measure. In Section 4 we evaluate c@1 in terms of stability, discrimination power and sensibil- ity, and some real examples of its behavior are given in the context of Question Answering. Related work is discussed in Section 5. 2 Looking for the Value of Not Responding Lets take the scenario of Reading Comprehension tests to argue about the development of the measure. Our scenario assumes …

Content truncated to 500 words