Automatic Selection of High Quality Parses Created By a Fully Unsupervised Parser

Proceedings of the Thirteenth Conference on Computational Natural Language Learning (CoNLL), pages 156–164, Boulder, Colorado, June 2009. c⃝2009 Association for Computational Linguistics Automatic Selection of High Quality Parses Created By a Fully Unsupervised Parser Roi Reichart ICNC The Hebrew University roiri@cs.huji.ac.il Ari Rappoport Institute of computer science The Hebrew University arir@cs.huji.ac.il Abstract The average results obtained by unsupervised statistical parsers have greatly improved in the last few years, but on many specific sentences they are of rather low quality. The output of such parsers is becoming valuable for vari- ous applications, and it is radically less expen- sive to create than manually annotated training data. Hence, automatic selection of high qual- ity parses created by unsupervised parsers is an important problem. In this paper we present PUPA, a POS-based Unsupervised Parse Assessment algorithm. The algorithm assesses the quality of a parse tree using POS sequence statistics collected from a batch of parsed sentences. We eval- uate the algorithm by using an unsupervised POS tagger and an unsupervised parser, se- lecting high quality parsed sentences from En- glish (WSJ) and German (NEGRA) corpora. We show that PUPA outperforms the leading previous parse assessment algorithm for su- pervised parsers, as well as a strong unsuper- vised baseline. Consequently, PUPA allows obtaining high quality parses without any hu- man involvement. 1 Introduction In unsupervised parsing an algorithm should un- cover the syntactic structure of an input sentence without using any manually created structural train- ing data. The last decade has seen significant progress in this field of research (Klein and Man- ning, 2002; Klein and Manning, 2004; Bod, 2006a; Bod, 2006b; Smith and Eisner, 2006; Seginer, 2007). Many NLP systems use the output of supervised parsers (e.g., (Kwok et al., 2001) for QA, (Moldovan et al., 2003) for IE, (Punyakanok et al., 2008) for SRL, (Srikumar et al., 2008) for Textual Inference and (Avramidis and Koehn, 2008) for MT). To achieve good performance, these parsers should be trained on large amounts of manually created train- ing data from a domain similar to that of the sen- tences they parse (Lease and Charniak, 2005; Mc- Closky and Charniak, 2008). In the highly variable Web, where many of these systems are used, it is very difficult to create a representative corpus for manual annotation. The high cost of manual annota- tion of training data for supervised parsers imposes a significant burden on their usage. A possible answer to this problem can be pro- vided by high quality parses produced by unsuper- vised parsers that require little to no manual efforts for their training. These parses can be used either as input for applications, or as training material for modern supervised parsers whose output will in turn be used by applications. Although unsupervised parser results improve, the quality of many of the parses they produce is still too low for such goals. For example, the Seginer (2007) parser achieves an F-score of 75.9% on the WSJ10 corpus and 59% on the NEGRA10 corpus, but the percentage …

Content truncated to 500 words