Complementarity, F-score, and NLP Evaluation

Complementarity, F-score, and NLP Evaluation Leon Derczynski University of Sheffield S1 4DP, UK leon.derczynski@sheffield.ac.uk Abstract This paper addresses the problem of quantifying the differences between entity extraction systems, where in general only a small proportion a document should be selected. Comparing overall accuracy is not very useful in these cases, as small differences in accuracy may correspond to huge differences in selections over the target minority class. Conventionally, one may use per-token complementarity to describe these differences, but it is not very useful when the set is heavily skewed. In such situations, which are common in information retrieval and entity recognition, metrics like precision and recall are typically used to describe performance. However, precision and recall fail to describe the differences between sets of objects selected by different decision strategies, instead just describing the proportional amount of correct and incorrect objects selected. This paper presents a method for measuring complementarity for precision, recall and F-score, quantifying the difference between entity extraction approaches. Keywords: Evaluation, F-Score, Ensembles 1. Introduction Many NLP systems are evaluated using F-score, which de- scribes system performance using a scale from zero to one. However, F-score lacks detail. When two approaches to an NLP problem achieve similar F-scores, they are not nec- essarily successful at the same kind of thing. This paper attempts to address this lack of detail, describing some of its impacts and putting forward an evaluation measure to be used alongside F-score that alleviates some of these prob- lems. F-score itself is derived from two summary measures: pre- cision and recall. Precision describes the proportion of en- tities – e.g. mentions of people, events, or any given target phenomenon – which a system returns that are correct. Re- call describes the proportion of all entities that potentially should be found, that a given system actually returns. Like F-score, these two are also summary measures, and so suf- fer from a similar lack of detail. When one NLP system achieves higher recall or precision than another, it does not imply that the better-scoring system accurately reproduces the results of the other and then exceeds them; rather, just the overall selection of entities is more precise, or more comprehensive, in some way. And indeed, high precision or recall can be achieved superficially through extreme – and often useless – conditions, like “return everything”. None of these measures can tell us much about the strate- gies that systems adopt. Importantly, they also cannot tell us about the differences in mistakes that systems make. Identifying strongly-different systems tells us which com- parisons are interesting to make, and which data we should look at in order to learn the most about a system, task or dataset. Seeing the impact that different strategies have can be interesting from a qualitative point of view: one system may do a lot better on data that is rich in a particular kind of linguistic phenomenon, for example. Also, it is interesting to see the impact that different kinds of prior knowledge and models …

Content truncated to 500 words