Semi-supervised Learning of Dependency Parsers using Generalized Expectation Criteria

Proceedings of the 47th Annual Meeting of the ACL and the 4th IJCNLP of the AFNLP, pages 360–368, Suntec, Singapore, 2-7 August 2009. c⃝2009 ACL and AFNLP Semi-supervised Learning of Dependency Parsers using Generalized Expectation Criteria Gregory Druck Dept. of Computer Science University of Massachusetts Amherst, MA 01003 gdruck@cs.umass.edu Gideon Mann Google, Inc. 76 9th Ave. New York, NY 10011 gideon.mann@gmail.com Andrew McCallum Dept. of Computer Science University of Massachusetts Amherst, MA 01003 mccallum@cs.umass.edu Abstract In this paper, we propose a novel method for semi-supervised learning of non- projective log-linear dependency parsers using directly expressed linguistic prior knowledge (e.g. a noun’s parent is often a verb). Model parameters are estimated us- ing a generalized expectation (GE) objec- tive function that penalizes the mismatch between model predictions and linguistic expectation constraints. In a comparison with two prominent “unsupervised” learn- ing methods that require indirect biasing toward the correct syntactic structure, we show that GE can attain better accuracy with as few as 20 intuitive constraints. We also present positive experimental results on longer sentences in multiple languages. 1 Introduction Early approaches to parsing assumed a grammar provided by human experts (Quirk et al., 1985). Later approaches avoided grammar writing by learning the grammar from sentences explicitly annotated with their syntactic structure (Black et al., 1992). While such supervised approaches have yielded accurate parsers (Charniak, 2001), the syntactic annotation of corpora such as the Penn Treebank is extremely costly, and consequently there are few treebanks of comparable size. As a result, there has been recent interest in unsupervised parsing. However, in order to at- tain reasonable accuracy, these methods have to be carefully biased towards the desired syntac- tic structure. This weak supervision has been encoded using priors and initializations (Klein and Manning, 2004; Smith, 2006), specialized models (Klein and Manning, 2004; Seginer, 2007; Bod, 2006), and implicit negative evi- dence (Smith, 2006). These indirect methods for leveraging prior knowledge can be cumbersome and unintuitive for a non-machine-learning expert. This paper proposes a method for directly guid- ing the learning of dependency parsers with nat- urally encoded linguistic insights. Generalized expectation (GE) (Mann and McCallum, 2008; Druck et al., 2008) is a recently proposed frame- work for incorporating prior knowledge into the learning of conditional random fields (CRFs) (Laf- ferty et al., 2001). GE criteria express a preference on the value of a model expectation. For example, we know that “in English, when a determiner is di- rectly to the left of a noun, the noun is usually the parent of the determiner”. With GE we may add a term to the objective function that encourages a feature-rich CRF to match this expectation on un- labeled data, and in the process learn about related features. In this paper we use a non-projective de- pendency tree CRF (Smith and Smith, 2007). While a complete exploration of linguistic prior knowledge for dependency parsing is beyond the scope of this paper, we provide several promis- ing demonstrations of the proposed method. On the English WSJ10 …

Content truncated to 500 words