Some Languages Seem Easier to Parse Because Their Treebanks Leak

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 2765–2770, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 2765 Some Languages Seem Easier to Parse Because Their Treebanks Leak Anders Søgaard Dpt. of Computer Science University of Copenhagen soegaard@di.ku.dk Abstract Cross-language differences in dependency parsing performance are mostly attributed to treebank size, average sentence length, aver- age dependency length, morphological com- plexity, and domain differences. In this paper I point to a factor not previously discussed: If we abstract away from words and dependency labels, how many graphs in the test data were seen in the training data? I discuss how to com- pute graph isomorphisms, and show that, tree- bank size aside, overlap between training and test graphs explains more of the observed vari- ation than standard explanations such as the above. 1 Introduction The state of the art in dependency parsing varies a lot across languages: on Polish, the best system in the CoNLL 2018 shared task achieved a labeled attachment score of 94.9% on held-out data; on Basque, the same number was 19.5%. Just a few years ago, a major source of variation was the com- plexity of the annotation schemes used in the dif- ferent treebanks; with the Universal Dependencies project,1 treebanks now follow the same annotation guidelines, but nevertheless, these performance dif- ferences persist.2 Differences are typically attributed to training set size (Vania et al., 2019), linguistic variation 1https://universaldependencies.org/ 2While Universal Dependencices have made the available dependency treebanks more compatible, treebanks were of course developed using very different protocols; some are automatically or semi-automatically converted from other formalisms, others written with the Universal Dependencies guidelines in mind; some, again, were developed by big teams, some by a single person. While protocol is hard to isolate and study – and while protocol may correlate both positively or negatively with parsing performance, i.e., it is easy to imagine a poorly designed treebank that is easy to parse – the proto- col likely has a significant downstream effect on performance; which means we can only hope to explain some of the variance in the experiments below. Yours drove responsively . 4 It is hers . 4 Figure 1: Isomorphic examples from UD-English- Pronouns. Left: Yours drove responsibly. Right: It is hers. The two sentences are associated with the same unlabeled directed graphs. (Nivre et al., 2007), sentence length or average gold dependency length (in the test data) (McDonald and Nivre, 2011), and domain differences between training and test data (Foster et al., 2011). Train- ing set size is undoubtedly a very strong predictor of parsing performance, but in this paper, overlap between unlabeled graphs in the training and test sections of a treebank is shown to be more pre- dictive than any of the other factors. Specifically, we compute equivalence classes over unlabeled de- pendency graphs – directed or undirected – and compute the ratio of trees in the treebanks’ test sections that are isomorphic to graphs observed in the training section, i.e., the …

Content truncated to 500 words