Evaluating Unsupervised Part-of-Speech Tagging for Grammar Induction

Proceedings of the 22nd International Conference on Computational Linguistics (Coling 2008), pages 329–336 Manchester, August 2008 Evaluating Unsupervised Part-of-Speech Tagging for Grammar Induction William P. Headden III, David McClosky, Eugene Charniak Brown Laboratory for Linguistic Information Processing (BLLIP) Brown University Providence, RI 02912 {headdenw,dmcc,ec}@cs.brown.edu Abstract This paper explores the relationship be- tween various measures of unsupervised part-of-speech tag induction and the per- formance of both supervised and unsuper- vised parsing models trained on induced tags. We find that no standard tagging metrics correlate well with unsupervised parsing performance, and several metrics grounded in information theory have no strong relationship with even supervised parsing performance. 1 Introduction There has been a great deal of recent interest in the unsupervised discovery of syntactic structure from text, both parts-of-speech (Johnson, 2007; Goldwater and Griffiths, 2007; Biemann, 2006; Dasgupta and Ng, 2007) and deeper grammatical structure like constituency and dependency trees (Klein and Manning, 2004; Smith, 2006; Bod, 2006; Seginer, 2007; Van Zaanen, 2001). While some grammar induction systems operate on raw text, many of the most successful ones presume prior part-of-speech tagging. Meanwhile, most re- cent work in part-of-speech induction focuses on increasing the degree to which their tags match hand-annotated ones such as those in the Penn Treebank. In this work our goal is to evaluate how im- provements in part-of-speech tag induction affects grammar induction. Using several different unsu- pervised taggers, we induce tags and train three grammar induction systems on the results. c⃝ 2008. Licensed under the Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported li- cense (http://creativecommons.org/licenses/by-nc-sa/3.0/). Some rights reserved. We then explore the relationship between the performance on common unsupervised tagging metrics and the performance of resulting grammar induction systems. Disconcertingly we find that they bear little to no relationship. This paper is organized as follows. In Section 2 we discuss unsupervised part-of-speech induction systems and common methods of evaluation. In Section 3, we describe grammar induction in gen- eral and discuss the systems with which we evalu- ate taggings. We present our experiments in Sec- tion 4, and finally conclude in Section 5. 2 Part-of-speech Tag Induction Part-of-speech tag induction can be thought of as a clustering problem where, given a corpus of words, we aim to group word tokens into syntactic classes. Two tasks are commonly labeled unsupervised part-of-speech induction. In the first, tag induction systems are allowed the use of a tagging dictionary, which specifies for each word a set of possible parts-of-speech (Merialdo, 1994; Smith and Eis- ner, 2005; Goldwater and Griffiths, 2007). In the second, only the word tokens and sentence bound- aries are given. In this work we focus on this latter task to explore grammar induction in a maximally unsupervised context. Tag induction systems typically focus on two sorts of features: distributional and morphologi- cal. Distributional refers to what sorts of words appear in close proximity to the word in question, while morphological refers to modeling the inter- nal structure of a word. All the systems below make use of distributional information, whereas …

Content truncated to 500 words