A method for in-depth comparative evaluation: How (dis)similar are outputs of pos taggers, dependency parsers and coreference resolvers really?

Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 188–198, Valencia, Spain, April 3-7, 2017. c⃝2017 Association for Computational Linguistics A method for in-depth comparative evaluation: How (dis)similar are outputs of POS taggers, dependency parsers and coreference resolvers really? Don Tuggener University of Zurich Institute of Computational Linguistics tuggener@cl.uzh.ch Abstract This paper proposes a generic method for the comparative evaluation of system out- puts. The approach is able to quantify the pairwise differences between two outputs and to unravel in detail what the differ- ences consist of. We apply our approach to three tasks in Computational Linguistics, i.e. POS tagging, dependency parsing, and coreference resolution. We find that sys- tem outputs are more distinct than the (of- ten) small differences in evaluation scores seem to suggest. 1 Introduction While there exist well-defined procedures to eval- uate system outputs against manually annotated gold data for many tasks in Computational Lin- guistics, generally little effort and exploration goes into identifying and analysing the differences between the outputs themselves. System outputs are usually compared in the following manner: The standard evaluation protocol for many tasks consists of comparing a system output (the re- sponse) to a manual annotation of the same data (the key). The difference between the response and key is quantified by a similarity metric such as ac- curacy, and different system outputs are compared to each other by ranking their scores with respect to the similarity metric. However, comparing the scores of the similarity metric does not paint the full picture of the differ- ences between the outputs, as we will demonstrate. There are hardly any principled or generic evalua- tion approaches that aim at comparing two or more system responses directly to investigate, highlight, and quantify their differences in detail. Closing this gap is desirable, because progress in many NLP tasks is often made in small steps, and it is often left unclear what the specific contribution of a novel approach is if the comparison to related work is solely based on a (sometimes marginally) small improvement in F1 score or accuracy. Fur- thermore, an overall improvement regarding accu- racy achieved by a new approach might come at the cost of failing in some areas where a baseline system was correct. Vice versa, a new approach might not improve overall accuracy, but solve par- ticular problems that no other system has been able to address. We propose an evaluation approach which aims at shedding light on the particular differences be- tween system responses and which is intended as a complement to evaluation metrics such as F1- score and accuracy. By doing so, we strive to pro- vide researchers with a tool that is able to give in- sight into the particular strengths and weaknesses of their system in comparison to others.1 Our method is also useful in iterative system develop- ment, as it tracks changes in the outputs of differ- ent system versions or feature sets. Furthermore, our …

Content truncated to 500 words