How Bad are PoS Tagger in Cross-Corpora Settings? Evaluating Annotation Divergence in the UD Project.

Proceedings of NAACL-HLT 2019, pages 218–227 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 218 How Bad are PoS Taggers in Cross-Corpora Settings? Evaluating Annotation Divergence in the UD Project Guillaume Wisniewski and Franc¸ois Yvon LIMSI, CNRS, Univ. Paris-Sud, Université Paris-Saclay, F-91405 Orsay, France {guillaume.wisniewski,francois.yvon}@limsi.fr Abstract The performance of Part-of-Speech tagging varies significantly across the treebanks of the Universal Dependencies project. This work points out that these variations may result from divergences between the annotation of train and test sets. We show how the annotation va- riation principle, introduced by Dickinson and Meurers (2003) to automatically detect errors in gold standard, can be used to identify incon- sistencies between annotations; we also eva- luate their impact on prediction performance. 1 Introduction The performance of Part-of-Speech (PoS) tag- gers significantly degrades when they are applied to test sentences that depart from training data. To illustrate this claim, Table 1 reports the error rate achieved by our in-house PoS tagger on the dif- ferent combinations of train and test sets of the French treebanks of the Universal Dependencies (UD) project (Nivre et al., 2018). 1 It shows that depending on the train and test sets considered, the performance can vary by a factor of more than 25. Many studies (Foster, 2010; Plank et al., 2014) attribute this drop in accuracy to covariate shift (Shimodaira, 2000), characterizing the diffe- rences between domains by a change in the mar- ginal distribution p(x) of the input (e.g. increase of out-of-vocabulary words, missing capitaliza- tion, different usage of punctuation, etc), while as- suming that the conditional label distribution re- mains unaffected. This work adopts a different point of view : we believe that the variation in tagging performance is due to a dataset shift (Candela et al., 2009), i.e. a change in the joint distribution of the features and labels. We assume that this change mainly results 1. See Section 2 for details regarding our experimental setting from incoherences in the annotations between cor- pora or even within the same corpus. Indeed, en- suring inter-annotator agreement in PoS tagging is known to be a difficult task as annotation gui- delines are not always interpreted in a consistent manner (Marcus et al., 1993). For instance, Man- ning (2011) shows that many errors in the WSJ corpus are just mistakes rather than uncertainties or difficulties in the task; Table 2 reports some of these annotation divergences that can be found in UD project. The situation is naturally worse in cross-corpora settings, in which treebanks are an- notated by different laboratories or groups. The contribution of this paper is threefold : — we show that, as already pointed out by de Marneffe et al. (2017), the variation prin- ciple of Boyd et al. (2008) can be used to flag potential annotation discrepancies in the UD project. Building on this prin- ciple, we introduce, to evaluate the anno- tation consistency of a corpus, several me- thods and metrics that can be used, during the annotation …

Content truncated to 500 words