Neutralizing Linguistically Problematic Annotations in Unsupervised Dependency Parsing Evaluation

Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics, pages 663–672, Portland, Oregon, June 19-24, 2011. c⃝2011 Association for Computational Linguistics Neutralizing Linguistically Problematic Annotations in Unsupervised Dependency Parsing Evaluation Roy Schwartz1 Omri Abend1∗ Roi Reichart2 Ari Rappoport1 1Institute of Computer Science Hebrew University of Jerusalem {roys02|omria01|arir}@cs.huji.ac.il 2Computer Science and Artificial Intelligence Laboratory Massachusetts Institute of Technology roiri@csail.mit.edu Abstract Dependency parsing is a central NLP task. In this paper we show that the common eval- uation for unsupervised dependency parsing is highly sensitive to problematic annotations. We show that for three leading unsupervised parsers (Klein and Manning, 2004; Cohen and Smith, 2009; Spitkovsky et al., 2010a), a small set of parameters can be found whose mod- ification yields a significant improvement in standard evaluation measures. These param- eters correspond to local cases where no lin- guistic consensus exists as to the proper gold annotation. Therefore, the standard evaluation does not provide a true indication of algorithm quality. We present a new measure, Neutral Edge Direction (NED), and show that it greatly reduces this undesired phenomenon. 1 Introduction Unsupervised induction of dependency parsers is a major NLP task that attracts a substantial amount of research (Klein and Manning, 2004; Cohen et al., 2008; Headden et al., 2009; Spitkovsky et al., 2010a; Gillenwater et al., 2010; Berg-Kirkpatrick et al., 2010; Blunsom and Cohn, 2010, inter alia). Parser quality is usually evaluated by comparing its output to a gold standard whose annotations are lin- guistically motivated. However, there are cases in which there is no linguistic consensus as to what the correct annotation is (K¨ubler et al., 2009). Examples include which verb is the head in a verb group struc- ture (e.g., “can” or “eat” in “can eat”), and which ∗ Omri Abend is grateful to the Azrieli Foundation for the award of an Azrieli Fellowship. noun is the head in a sequence of proper nouns (e.g., “John” or “Doe” in “John Doe”). We refer to such annotations as (linguistically) problematic. For such cases, evaluation measures should not punish the al- gorithm for deviating from the gold standard. In this paper we show that the evaluation mea- sures reported in current works are highly sensitive to the annotation in problematic cases, and propose a simple new measure that greatly neutralizes the problem. We start from the following observation: for three leading algorithms (Klein and Manning, 2004; Co- hen and Smith, 2009; Spitkovsky et al., 2010a), a small set (at most 18 out of a few thousands) of pa- rameters can be found whose modification dramati- cally improves the standard evaluation measures (the attachment score measure by 9.3-15.1%, and the undirected measure by a smaller but still significant 1.3-7.7%). The phenomenon is implementation in- dependent, occurring with several algorithms based on a fundamental probabilistic dependency model1. We show that these parameter changes can be mapped to edge direction changes in local structures in the dependency graph, and that these correspond to problematic annotations. Thus, the standard eval- uation measures do not …

Content truncated to 500 words