Quantifying training challenges of dependency parsers

Proceedings of the 27th International Conference on Computational Linguistics, pages 3191–3202 Santa Fe, New Mexico, USA, August 20-26, 2018. 3191 Quantifying training challenges of dependency parsers Lauriane Aufrant1,2, Guillaume Wisniewski1 and Franc¸ois Yvon1 1LIMSI, CNRS, Univ. Paris-Sud, Universit´e Paris-Saclay, 91 405 Orsay, France 2DGA, 60 boulevard du G´en´eral Martial Valin, 75 509 Paris, France {lauriane.aufrant,guillaume.wisniewski,francois.yvon}@limsi.fr Abstract Not all dependencies are equal when training a dependency parser: some are straightforward enough to be learned with only a sample of data, others embed more complexity. This work introduces a series of metrics to quantify those differences, and thereby to expose the shortcom- ings of various parsing algorithms and strategies. Apart from a more thorough comparison of parsing systems, these new tools also prove useful for characterizing the information conveyed by cross-lingual parsers, in a quantitative but still interpretable way. 1 Introduction 10 annotated sentences suffice to train a dependency parser that can correctly predict around 70% of the dependencies in French and Italian, but also in Japanese, Greek, Urdu and other languages. This surprising fact raises questions on how to interpret such numbers: what do those 70% dependencies look like? Are they evenly distributed or do they correspond to specific dependency structures? This work is based on the intuition that in any treebank, some dependencies are intuitively easy to learn (typically nearly deterministic attachments like the dependencies between a noun and its determiner), while others seem much harder to predict, corresponding to complex semantic attachments or long lists of exceptions in the underlying grammar of the language. We thus propose one way to investigate that distinction, by evaluating the amount of training material needed to learn each kind of dependencies: with that viewpoint, a dependency structure is qualified as ‘easy to learn’ when it can be learned from only a few examples and as ‘difficult to learn’ otherwise. In this paper, we formalize this idea and propose empirical ways to measure the difficulty to learn certain dependencies. Our metric proposal can be useful as an analysis tool for comparing existing parsers, which we illus- trate both in monolingual and cross-lingual settings, but also to gain some insight on the information reliably conveyed by parsers trained on a few sentences. Ulinski et al. (2016) have indeed shown that such parsers can be successfully leveraged to speed up the manual annotation process. This observa- tion confirms that they provide some useful content to the annotators, which our formalism can help characterizing beforehand. The rest of this paper is organized as follows. Section 2 presents the dependency parsing task and the parsers used in this study. Our method for estimating difficulty is introduced in Section 3. We then apply these newly designed tools to conduct two analyses: a detailed comparison of several state-of-the-art parsers (Section 4), as well as a fine-grained evaluation of cross-lingual transfer (Section 5). 2 Dependency parsing The purpose of the dependency parsing task is to predict, for each token in a sentence (the child), the token it depends on (its head), …

Content truncated to 500 words