Two Decades of Unsupervised POS Induction: How Far Have We Come?

Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pages 575–584, MIT, Massachusetts, USA, 9-11 October 2010. c⃝2010 Association for Computational Linguistics Two Decades of Unsupervised POS induction: How far have we come? Christos Christodoulopoulos School of Informatics University of Edinburgh christos.c@ed.ac.uk Sharon Goldwater School of Informatics University of Edinburgh sgwater@inf.ed.ac.uk Mark Steedman School of Informatics University of Edinburgh steedman@inf.ed.ac.uk Abstract Part-of-speech (POS) induction is one of the most popular tasks in research on unsuper- vised NLP. Many different methods have been proposed, yet comparisons are difficult to make since there is little consensus on eval- uation framework, and many papers evalu- ate against only one or two competitor sys- tems. Here we evaluate seven different POS induction systems spanning nearly 20 years of work, using a variety of measures. We show that some of the oldest (and simplest) systems stand up surprisingly well against more recent approaches. Since most of these systems were developed and tested using data from the WSJ corpus, we compare their generalization abil- ities by testing on both WSJ and the multi- lingual Multext-East corpus. Finally, we in- troduce the idea of evaluating systems based on their ability to produce cluster prototypes that are useful as input to a prototype-driven learner. In most cases, the prototype-driven learner outperforms the unsupervised system used to initialize it, yielding state-of-the-art results on WSJ and improvements on non- English corpora. 1 Introduction In recent years, unsupervised learning has become a hot area in NLP, in large part due to the use of sophisticated machine learning approaches which promise to deliver better results than more tradi- tional methods. Often the new approaches are tested using part-of-speech (POS) tagging as an example application, and usually they are shown to perform better than one or another comparison system. How- ever, it is difficult to draw overall conclusions about the relative performance of unsupervised POS tag- ging systems because of differences in evaluation measures, and the fact that no paper includes di- rect comparisons against more than a few other sys- tems. In this paper, we attempt to remedy that situation by providing a comprehensive evaluation of seven different POS induction systems spanning nearly 20 years of research. We focus specifically on POS induction systems, where no prior knowl- edge is available, in contrast to POS disambigua- tion systems (Merialdo, 1994; Toutanova and John- son, 2007; Naseem et al., 2009; Ravi and Knight, 2009; Smith and Eisner, 2005), which use a dic- tionary to provide possible tags for some or all of the words in the corpus, or prototype-driven sys- tems (Haghighi and Klein, 2006), which use a small set of prototypes for each tag class, but no dictio- nary. Our motivation stems from another part of our own research, in which we are trying to use NLP systems on over 50 low-density languages (some of them dead) where both tagged corpora and language speakers are mostly unavailable. We therefore de- sire to use these systems straight out of the …

Content truncated to 500 words