Learnability-Based Syntactic Annotation Design

Proceedings of COLING 2012: Technical Papers, pages 2405–2422, COLING 2012, Mumbai, December 2012. Learnability-based Syntactic Annotation Design Roy Schwartz Omri Abend Ari Rappoport Institute of Computer Science, Hebrew University of Jerusalem {roys02|omria01|arir}@cs.huji.ac.il Abstract There is often more than one way to represent syntactic structures, even within a given formalism. Selecting one representation over another may affect parsing performance. Therefore, selecting between alternative syntactic representations (henceforth, syntactic selection) is an essential step in designing an annotation scheme. We present a methodology for syntactic selection and apply it to six central dependency structures. Our methodology compares pairs of annotation schemes that differ in the annotation of a single structure. It selects the more learnable scheme, namely the one that can be better learned using statistical parsers. We find that in three of the structures, one anno- tation is unequivocally better than the alternatives. Our results are consistent over various settings involving five parsers and two definitions of learnability. Furthermore, we show that the learnabil- ity gains incurred by our selections are both considerable (error reductions of up to 19.8%) and additive. The contribution of this work is in demonstrating that syntactic selection has a substantial and predictable effect on parsing performance, and showing that this effect can be effectively used in designing syntactic annotation schemes. Keywords: Syntactic annotation design, Learnability, Parsing. 2405 1 Introduction The formal manner in which syntactic relations are represented is at the core of the study of gram- mar. Numerous representations have been proposed over the years for expressing similar syntactic relations. This diversity of representations is expressed in a variety of syntactic annotation schemes currently in use in NLP. Examples include, for constituency annotation, schemes by (Marcus et al. 1993; Sampson 1995; Nelson et al. 2002, inter alia) and for dependency annotation, schemes by (Collins 1999; Rambow et al. 2002; Yamada and Matsumoto 2003; Johansson and Nugues 2007, inter alia). Variation within the same formalism is expressed in structures that have several alterna- tive annotations (henceforth Varying Syntactic Structure or VSS). In this work we focus on dependency structures, where some of the most basic structures are VSS’s. One example is prepositional phrases, which consist of a preposition followed by a noun phrase (e.g., “about everyone”). While some schemes select the preposition to head the NP (Collins 1999), others select the NP as the head of the preposition (Johansson and Nugues 2007) (see Fig- ure 1). Other prominent VSS’s include coordination structures and verb group constructions (see Section 3). In fact, more than 40% of the tokens in the Penn Treebank (Marcus et al. 1993) partici- pate in at least one VSS (Schwartz et al. 2011). IN about NN everyone Figure 1: An example of a prepositional phrase – a Varying Syntactic Structure (VSS). Both annotation alternatives for this structure are plausible: either setting the preposition (“about” – solid line) as head, or the noun phrase (“everyone” – dashed line). Despite the similar content represented by the alternative annotations to VSS’s, selecting one over the other (syntactic …

Content truncated to 500 words