A Bayesian Mixture Model for PoS Induction Using Multiple Features

Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 638–647, Edinburgh, Scotland, UK, July 27–31, 2011. c⃝2011 Association for Computational Linguistics A Bayesian Mixture Model for Part-of-Speech Induction Using Multiple Features Christos Christodoulopoulos School of Informatics University of Edinburgh christos.c@ed.ac.uk Sharon Goldwater School of Informatics University of Edinburgh sgwater@inf.ed.ac.uk Mark Steedman School of Informatics University of Edinburgh steedman@inf.ed.ac.uk Abstract In this paper we present a fully unsupervised syntactic class induction system formulated as a Bayesian multinomial mixture model, where each word type is constrained to belong to a single class. By using a mixture model rather than a sequence model (e.g., HMM), we are able to easily add multiple kinds of features, including those at both the type level (mor- phology features) and token level (context and alignment features, the latter from parallel cor- pora). Using only context features, our sys- tem yields results comparable to state-of-the art, far better than a similar model without the one-class-per-type constraint. Using the addi- tional features provides added benefit, and our final system outperforms the best published results on most of the 25 corpora tested. 1 Introduction Research on unsupervised learning for NLP has be- come widespread recently, with part-of-speech in- duction, or syntactic class induction, being a partic- ularly popular task.1 However, despite a recent pro- liferation of syntactic class induction systems (Bie- mann, 2006; Goldwater and Griffiths, 2007; John- son, 2007; Ravi and Knight, 2009; Berg-Kirkpatrick et al., 2010; Lee et al., 2010), careful compari- son indicates that very few systems perform better than some much simpler and quicker methods dating back ten or even twenty years (Christodoulopoulos 1The task is more commonly referred to as part-of-speech induction, but we prefer the term syntactic class induction since the induced classes may not coincide with part-of-speech tags. et al., 2010). This fact suggests that we should con- sider which features of the older systems led to their success, and attempt to combine these features with some of the machine learning methods introduced by the more recent systems. We pursue this strat- egy here, developing a system based on Bayesian methods where the probabilistic model incorporates several insights from previous work. Perhaps the most important property of our model is that it is type-based, meaning that all tokens of a given word type are assigned to the same clus- ter. This property is not strictly true of linguistic data, but is a good approximation: as Lee et al. (2010) note, assigning each word type to its most frequent part of speech yields an upper bound ac- curacy of 93% or more for most languages. Since this is much better than the performance of cur- rent unsupervised syntactic class induction systems, constraining the model in this way seems likely to improve performance by reducing the number of parameters in the model and incorporating useful linguistic knowledge. Both of the older systems discussed by Christodoulopoulos et al. (2010), i.e., Clark (2003) and Brown et al. (1992), included this constraint and achieved …

Content truncated to 500 words