Simple Semi-supervised Dependency Parsing

Proceedings of ACL-08: HLT, pages 595–603, Columbus, Ohio, USA, June 2008. c⃝2008 Association for Computational Linguistics Simple Semi-supervised Dependency Parsing Terry Koo, Xavier Carreras, and Michael Collins MIT CSAIL, Cambridge, MA 02139, USA {maestro,carreras,mcollins}@csail.mit.edu Abstract We present a simple and effective semi- supervised method for training dependency parsers. We focus on the problem of lex- ical representation, introducing features that incorporate word clusters derived from a large unannotated corpus. We demonstrate the ef- fectiveness of the approach in a series of de- pendency parsing experiments on the Penn Treebank and Prague Dependency Treebank, and we show that the cluster-based features yield substantial gains in performance across a wide range of conditions. For example, in the case of English unlabeled second-order parsing, we improve from a baseline accu- racy of 92.02% to 93.16%, and in the case of Czech unlabeled second-order parsing, we improve from a baseline accuracy of 86.13% to 87.13%. In addition, we demonstrate that our method also improves performance when small amounts of training data are available, and can roughly halve the amount of super- vised data required to reach a desired level of performance. 1 Introduction In natural language parsing, lexical information is seen as crucial to resolving ambiguous relationships, yet lexicalized statistics are sparse and difficult to es- timate directly. It is therefore attractive to consider intermediate entities which exist at a coarser level than the words themselves, yet capture the informa- tion necessary to resolve the relevant ambiguities. In this paper, we introduce lexical intermediaries via a simple two-stage semi-supervised approach. First, we use a large unannotated corpus to define word clusters, and then we use that clustering to construct a new cluster-based feature mapping for a discriminative learner. We are thus relying on the ability of discriminative learning methods to identify and exploit informative features while remaining ag- nostic as to the origin of such features. To demon- strate the effectiveness of our approach, we conduct experiments in dependency parsing, which has been the focus of much recent research—e.g., see work in the CoNLL shared tasks on dependency parsing (Buchholz and Marsi, 2006; Nivre et al., 2007). The idea of combining word clusters with dis- criminative learning has been previously explored by Miller et al. (2004), in the context of named- entity recognition, and their work directly inspired our research. However, our target task of depen- dency parsing involves more complex structured re- lationships than named-entity tagging; moreover, it is not at all clear that word clusters should have any relevance to syntactic structure. Nevertheless, our experiments demonstrate that word clusters can be quite effective in dependency parsing applications. In general, semi-supervised learning can be mo- tivated by two concerns: first, given a fixed amount of supervised data, we might wish to leverage ad- ditional unlabeled data to facilitate the utilization of the supervised corpus, increasing the performance of the model in absolute terms. Second, given a fixed target performance level, we might wish to use un- labeled data to reduce the amount of …

Content truncated to 500 words