Learning part-of-speech taggers with inter-annotator agreement loss

Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, pages 742–751, Gothenburg, Sweden, April 26-30 2014. c⃝2014 Association for Computational Linguistics Learning part-of-speech taggers with inter-annotator agreement loss Barbara Plank, Dirk Hovy, Anders Søgaard Center for Language Technology University of Copenhagen, Denmark Njalsgade 140, DK-2300 Copenhagen S bplank@cst.dk,dirk@cst.dk,soegaard@hum.ku.dk Abstract In natural language processing (NLP) an- notation projects, we use inter-annotator agreement measures and annotation guide- lines to ensure consistent annotations. However, annotation guidelines often make linguistically debatable and even somewhat arbitrary decisions, and inter- annotator agreement is often less than perfect. While annotation projects usu- ally specify how to deal with linguisti- cally debatable phenomena, annotator dis- agreements typically still stem from these “hard” cases. This indicates that some er- rors are more debatable than others. In this paper, we use small samples of doubly- annotated part-of-speech (POS) data for Twitter to estimate annotation reliability and show how those metrics of likely inter- annotator agreement can be implemented in the loss functions of POS taggers. We find that these cost-sensitive algorithms perform better across annotation projects and, more surprisingly, even on data an- notated according to the same guidelines. Finally, we show that POS tagging mod- els sensitive to inter-annotator agreement perform better on the downstream task of chunking. 1 Introduction POS-annotated corpora and treebanks are collec- tions of sentences analyzed by linguists accord- ing to some linguistic theory. The specific choice of linguistic theory has dramatic effects on down- stream performance in NLP tasks that rely on syn- tactic features (Elming et al., 2013). Variation across annotated corpora in linguistic theory also poses challenges to intrinsic evaluation (Schwartz et al., 2011; Tsarfaty et al., 2012), as well as for languages where available resources are mu- tually inconsistent (Johansson, 2013). Unfortu- nately, there is no grand unifying linguistic the- ory of how to analyze the structure of sentences. While linguists agree on certain things, there is still a wide range of unresolved questions. Con- sider the following sentence: (1) @GaryMurphyDCU of @DemMattersIRL will take part in a panel discussion on Octo- ber 10th re the aftermath of #seanref ... While linguists will agree that in is a preposi- tion, and panel discussion a compound noun, they are likely to disagree whether will is heading the main verb take or vice versa. Even at a more basic level of analysis, it is not completely clear how to assign POS tags to each word in this sentence: is part a particle or a noun; is 10th a numeral or a noun? Some linguistic controversies may be resolved by changing the vocabulary of linguistic theory, e.g., by leaving out numerals or introducing ad hoc parts of speech, e.g. for English to (Marcus et al., 1993) or words ending in -ing (Manning, 2011). However, standardized label sets have practical advantages in NLP (Zeman and Resnik, 2008; Zeman, 2010; Das and Petrov, 2011; Petrov et al., 2012; McDonald et al., 2013). For these and other reasons, our annotators (even …

Content truncated to 500 words