Do dependency parsing metrics correlate with human judgments?

Proceedings of the 19th Conference on Computational Language Learning, pages 315–320, Beijing, China, July 30-31, 2015. c⃝2015 Association for Computational Linguistics Do dependency parsing metrics correlate with human judgments? Barbara Plank,⋆ H´ector Mart´ınez Alonso,⋆ ˇZeljko Agi´c,⋆ Danijela Merkler,† Anders Søgaard⋆ ⋆Center for Language Technology, University of Copenhagen, Denmark †Department of Linguistics, University of Zagreb, Croatia bplank@cst.dk Abstract Using automatic measures such as labeled and unlabeled attachment scores is com- mon practice in dependency parser evalu- ation. In this paper, we examine whether these measures correlate with human judg- ments of overall parse quality. We ask lin- guists with experience in dependency an- notation to judge system outputs. We mea- sure the correlation between their judg- ments and a range of parse evaluation met- rics across five languages. The human- metric correlation is lower for dependency parsing than for other NLP tasks. Also, inter-annotator agreement is sometimes higher than the agreement between judg- ments and metrics, indicating that the stan- dard metrics fail to capture certain aspects of parse quality, such as the relevance of root attachment or the relative importance of the different parts of speech. 1 Introduction In dependency parser evaluation, the standard ac- curacy metrics—labeled and unlabeled attachment scores—are defined simply as averages over cor- rect attachment decisions. Several authors have pointed out problems with these metrics; they are both sensitive to annotation guidelines (Schwartz et al., 2012; Tsarfaty et al., 2011), and they fail to say anything about how parsers fare on rare, but important linguistic constructions (Nivre et al., 2010). Both criticisms rely on the intuition that some parsing errors are more important than oth- ers, and that our metrics should somehow reflect that. There are sentences that are hard to anno- tate because they are ambiguous, or because they contain phenomena peripheral to linguistic theory, such as punctuation, clitics, or fragments. Man- ning (2011) discusses similar issues for part-of- speech tagging. To measure the variable relevance of parsing errors, we present experiments with human judg- ment of parse output quality across five languages: Croatian, Danish, English, German, and Spanish. For the human judgments, we asked professional linguists with dependency annotation experience to judge which of two parsers produced the bet- ter parse. Our stance here is that, insofar ex- perts are able to annotate dependency trees, they are also able to determine the quality of a pre- dicted syntactic structure, which we can in turn use to evaluate parser evaluation metrics. Even though downstream evaluation is critical in as- sessing the usefulness of parses, it also presents non-trivial challenges in choosing the appropriate downstream tasks (Elming et al., 2013), we see human judgments as an important supplement to extrinsic evaluation. To the best of our knowledge, no prior study has analyzed the correlation between dependency parsing metrics and human judgments. For a range of other NLP tasks, metrics have been evaluated by how well they correlate with human judgments. For instance, the standard automatic metrics for certain tasks—such as BLEU in machine trans- lation, or …

Content truncated to 500 words