Multi-lingual Dependency Parsing Evaluation: a Large-scale Analysis of Word Order Properties using Artificial Data

Multi-lingual Dependency Parsing Evaluation: a Large-scale Analysis of Word Order Properties using Artificial Data Kristina Gulordava and Paola Merlo Department of Linguistics University of Geneva 5 Rue de Candolle, CH-1211 Gen`eve 4 kristina.gulordava@unige.ch, paola.merlo@unige.ch Abstract The growing work in multi-lingual parsing faces the challenge of fair comparative eval- uation and performance analysis across lan- guages and their treebanks. The difficulty lies in teasing apart the properties of treebanks, such as their size or average sentence length, from those of the annotation scheme, and from the linguistic properties of languages. We pro- pose a method to evaluate the effects of word order of a language on dependency parsing performance, while controlling for confound- ing treebank properties. The method uses artificially-generated treebanks that are mini- mal permutations of actual treebanks with re- spect to two word order properties: word or- der variation and dependency lengths. Based on these artificial data on twelve languages, we show that longer dependencies and higher word order variability degrade parsing perfor- mance. Our method also extends to mini- mal pairs of individual sentences, leading to a finer-grained understanding of parsing errors. 1 Introduction Fair comparative performance evaluation across lan- guages and their treebanks is one of the difficul- ties for work on multi-lingual parsing (Buchholz and Marsi, 2006; Nivre et al., 2007; Seddah et al., 2011). The differences in parsing performance can be the result of disparate properties of treebanks (such as their size or average sentence length), choices in an- notation schemes, and the linguistic properties of languages. Despite recent attempts to create and apply cross-linguistic and cross-framework evalua- tion procedures (Tsarfaty et al., 2011; Seddah et al., 2013), there is no commonly used method of anal- ysis of parsing performance which accounts for dif- ferent linguistic and extra-linguistic factors of tree- banks and teases them apart. When investigating possible causal factors for ob- served phenomena, one powerful method, if avail- able, consists in intervening on the postulated causes to observe possible changes in the observed effects. In other words, if A causes B, then changing A or properties of A should result in an observable change in B. This interventionist approach to the study of causality creates counterfactual data and a type of controlled modification that is wide-spread in experimental methodology, but that is not widely used in fields that rely on observational data, such as corpus-driven natural language processing. In analyses of parsing performance, it is custom- ary to manipulate and control word-level features, such as part-of-speech tags or morphological fea- tures. These types of features can be easily omit- ted or modified to assess their contribution to pars- ing performance. However, higher-order features, such as linear word order precedence properties, are much harder to define and to manipulate. A parsing performance analysis based on controlled modifica- tion of word order, in fact, has not been reported pre- viously. We propose such a method based on word order permutations which allows us to manipulate word order properties analogously to familiar word- level properties and …

Content truncated to 500 words