Revisiting the Effects of Leakage on Dependency Parsing

Findings of the Association for Computational Linguistics: ACL 2022, pages 2925 - 2934 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Revisiting the Effects of Leakage on Dependency Parsing Nathaniel Krasner*,1, Miriam Wanner*,2, Antonios Anastasopoulos1 1Department of Computer Science, George Mason University 2Department of Computer Science, University of Virginia natekrasner@gmail.com, msw2vg@virginia.edu, antonis@gmu.edu Abstract Recent work by Søgaard (2020) showed that, treebank size aside, overlap between train- ing and test graphs (termed leakage) explains more of the observed variation in dependency parsing performance than other explanations. In this work we revisit this claim, testing it on more models and languages. We find that it only holds for zero-shot cross-lingual settings. We then propose a more fine-grained measure of such leakage which, unlike the original mea- sure, not only explains but also correlates with observed performance variation.1 1 Introduction Syntactic parsing has long been one of the core natural language processing (NLP) tasks, and the proliferation of the Universal Dependencies project (UD; de Marneffe et al., 2021; Nivre et al., 2017) has allowed the development and compari- son of monolingual and multilingual models under the same syntactic framework. The performance of the dependency parsers, however, varies wildly across languages, with state- of-the-art performance ranging from labeled attach- ment scores below 20 (e.g. for Amharic, Erzya, Komi, or Yoruba) to more than 90 (e.g. for Span- ish, Polish, Russian, or Greek). As the UD tree- banks follow mostly similar annotation guidelines, comparisons of the parsing performance across lan- guages are now possible, to an extent.2 In an effort to explain these cross-lingual per- formance differences, researchers have proposed treebank size (Vania et al., 2019), linguistic varia- tion (Nivre et al., 2007), test data sentence length * Equal contribution. Work performed at GMU. 1Code and data are available here: https://github.com/ miriamwanner/reu-nlp-project 2Different treebank creation protocols followed across lan- guages (whose effects are hard to isolate or measure) can be a significant source of variation. Nevertheless, some of the observed variation can be possibly explained by other factors. We direct the reader to footnote 2 of (Søgaard, 2020). Dependency Trees: She saw it PRON VERB PRON root nsubj dobj The big boat DET ADJ NOUN root det amod Søgaard (2020) Unlabeled Directed Graphs: Node- and Edge-Labeled Directed Graphs: rt V PR PR nsubj dobj rt N DT A det amod Figure 1: Only labeled reductions produce different graphs for these fundamentally different sentences. Un- der unlabeled leakage, the two trees do leak. When tak- ing labels into account the two trees belong to different isomorphisms and are not considered “leaky". or average gold dependency length (McDonald and Nivre, 2011), and domain differences between training and test data (Foster et al., 2011), as poten- tial predictors. Recently, Søgaard (2020) proposed that the proportion of isomorphic graph structures between the training and testing data (leakage) is a stronger predictor of the parsers’ performance than any of the previously listed attributes other than training treebank size. Søgaard (2020) concludes that “some languages seem easier to parse because their …

Content truncated to 500 words