Fine-Grained Error Analysis and Fair Evaluation of Labeled Spans

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 1400–1407 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 1400 Fine-Grained Error Analysis and Fair Evaluation of Labeled Spans Katrin Ortmann Department of Linguistics Fakultät für Philologie Ruhr-Universität Bochum ortmann@linguistics.rub.de Abstract The traditional evaluation of labeled spans with precision, recall, and F1-score has undesirable effects due to double penalties. Annotations with incorrect label or boundaries count as two errors instead of one, despite being closer to the target annotation than false positives or false negatives. In this paper, new error types are introduced, which more accurately reflect true anno- tation quality and ensure that every annotation counts only once. An algorithm for error identification in flat and multi-level annotations is presented and complemented with a proposal on how to calculate meaningful precision, recall, and F1-scores based on the more fine-grained error types. The exemplary application to three different annotation tasks (NER, chunking, parsing) shows that the suggested procedure not only prevents double penalties but also allows for a more detailed error analy- sis, thereby providing more insight into the actual weaknesses of a system. Keywords: Evaluation, F-score, labeled spans, double penalties, error analysis 1. Introduction Evaluation in NLP serves two main purposes: (i) deter- mining how good a system is at a given task and com- paring its performance to other systems, and (ii) ana- lyzing the errors a system makes to be able to improve it. For 1:1 mapping tasks like POS tagging, the proce- dure is clear-cut. Every token receives exactly one tag, and the number of correctly assigned tags is compared to the incorrect tags and reported as accuracy, possibly accompanied by a confusion matrix. For tasks that include the annotation of spans or do not require every token to receive an annotation, e.g., tok- enization, named entity recognition (NER), or chunk- ing, the incorrect annotations can be further divided into false positives (FP, i.e., superfluous annotations) and false negatives (FN, i.e., missing annotations). Sys- tem performance, in this case, is measured by com- paring the two types of incorrect annotations with the number of true positives (TP), and the results are re- ported as recall (Eq. 1; ‘how many annotations that should be present are actually there’) and precision (Eq. 2; ‘how many of the annotations that are present are actually correct’). Usually, there is a trade-off between precision and recall because improving one likely worsens the other. Recall = TP TP + FN (1) Precision = TP TP + FP (2) The harmonic mean of precision and recall, better known as F1-score, is consulted to compare different systems based on a single number (Eq. 3). F 1 = 2 ∗ Precision ∗ Recall Precision + Recall (3) There are certain issues with these evaluation metrics, though (cf., e.g., Shao et al. (2017) for the task of word tokenization). The focus of this paper will be on the yet unsolved problem of double penalties for the anno- tation …

Content truncated to 500 words