Learning to parse with IAA-weighted loss

Human Language Technologies: The 2015 Annual Conference of the North American Chapter of the ACL, pages 1357–1361, Denver, Colorado, May 31 – June 5, 2015. c⃝2015 Association for Computational Linguistics Learning to parse with IAA-weighted loss H´ector Mart´ınez Alonso† Barbara Plank† Arne Skjærholt‡ Anders Søgaard† †Njalsgade 140, Copenhagen (Denmark), University of Copenhagen ‡Gaustadall´een 23B, Oslo (Norway), University of Oslo alonso@hum.ku.dk,bplank@cst.dk,arnskj@ifi.uio.no,soegaard@hum.ku.dk Abstract Natural language processing (NLP) annota- tion projects employ guidelines to maximize inter-annotator agreement (IAA), and models are estimated assuming that there is one single ground truth. However, not all disagreement is noise, and in fact some of it may contain valu- able linguistic information. We integrate such information in the training of a cost-sensitive dependency parser. We introduce five differ- ent factorizations of IAA and the correspond- ing loss functions, and evaluate these across six different languages. We obtain robust im- provements across the board using a factoriza- tion that considers dependency labels and di- rectionality. The best method-dataset combi- nation reaches an average overall error reduc- tion of 6.4% in labeled attachment score. 1 Introduction Typically, NLP annotation projects employ guide- lines to maximize inter-annotator agreement. Pos- sible inconsistencies are resolved by adjudication, and models are induced assuming there is one sin- gle ground truth. However, there exist linguistically hard cases where there is no clear answer (Zeman, 2010; Manning, 2011), and incorporating such dis- agreements into the training of a model has proven helpful for POS tagging (Plank et al., 2014a; Plank et al., 2014b). Inter-annotator agreement (IAA) is straight- forward to calculate for POS, but not for depen- dency trees. There is no well-established standard for computing agreement on trees (Skjærholt, 2014). For a dependency tree, annotators can disagree in attachment, labeling, or both. We implement dif- ferent strategies, i.e., factorizations (§2), to capture disagreement on specific syntactic phenomena. Our hypothesis is that a dependency parser can be informed of disagreements to regularize over anno- tators’ biases. Testing our hypothesis requires the availability of doubly-annotated data, and involves two steps: i) how to factorize attachment or labeling disagreements; and ii) how to inform the parser of them during learning (§3). 2 Factorizations Assume a sample of sentences annotated by annota- tors A1 and A2. With such a sample we can estimate probabilities of the two annotators’ disagreeing on the annotation of a word or span, relative to some de- pendency tree factorization. We factorize disagree- ment on dependency tree annotations relative to four properties of the annotated dependency edges: the POS of the dependent, the POS of the head, the la- bel of the edge and the direction (left or right) of the head with regards to the dependent. This section describes the different factorizations. We present five factorizations, depicted in Fig- ure 1. With artificial root notes, all words in a depen- dency tree have one incoming edge. This means that in our sample, any word wi has two ⟨headId, label⟩ annotations, i.e., ⟨h1, l1⟩ and ⟨h2, l2⟩ given by A1 and A2, respectively, with …

Content truncated to 500 words