Crouching Dirichlet, Hidden Markov Model: Unsupervised POS Tagging with Context Local Tag Generation

Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pages 196–206, MIT, Massachusetts, USA, 9-11 October 2010. c⃝2010 Association for Computational Linguistics Crouching Dirichlet, Hidden Markov Model: Unsupervised POS Tagging with Context Local Tag Generation Taesun Moon, Katrin Erk, and Jason Baldridge Department of Linguistics University of Texas at Austin 1 University Station B5100 Austin, TX 78712-0198 USA {tsmoon,katrin.erk,jbaldrid}@mail.utexas.edu Abstract We define the crouching Dirichlet, hidden Markov model (CDHMM), an HMM for part- of-speech tagging which draws state prior dis- tributions for each local document context. This simple modification of the HMM takes advantage of the dichotomy in natural lan- guage between content and function words. In contrast, a standard HMM draws all prior dis- tributions once over all states and it is known to perform poorly in unsupervised and semi- supervised POS tagging. This modification significantly improves unsupervised POS tag- ging performance across several measures on five data sets for four languages. We also show that simply using different hyperparameter values for content and function word states in a standard HMM (which we call HMM+) is surprisingly effective. 1 Introduction Hidden Markov Models (HMMs) are simple, ver- satile, and widely-used generative sequence models. They have been applied to part-of-speech (POS) tag- ging in supervised (Brants, 2000), semi-supervised (Goldwater and Griffiths, 2007; Ravi and Knight, 2009) and unsupervised (Johnson, 2007) training scenarios. Though discriminative models achieve better performance in both semi-supervised (Smith and Eisner, 2005) and supervised (Toutanova et al., 2003) learning, there has been only limited work on unsupervised discriminative sequence models (e.g., on synthetic data and protein sequences (Xu et al., 2006)), and none to POS tagging. The tagging accuracy of purely unsupervised HMMs is far below that of supervised and semi- supervised HMMs; this is unsurprising as it is still not well understood what kind of structure is being found by an unconstrained HMM (Headden III et al., 2008). However, HMMs are fairly simple directed graphical models, and it is straightforward to ex- tend them to define alternative generative processes. This also applies to linguistically motivated HMMs for recovering states and sequences that correspond more closely to those implicitly defined by linguists when they label sentences with parts-of-speech. One way in which a basic HMM’s structure is a poor model for POS tagging is that there is no inher- ent distinction between (open-class) content words and (closed-class) function words. Here, we propose two extensions to the HMM. The first, HMM+, is a very simple modification where two different hyper- parameters are posited for content states and func- tion states, respectively. The other is the crouch- ing Dirichlet, hidden Markov model (CDHMM), an extended HMM that captures this dichotomy based on the statistical evidence that comes from context. Content states display greater variance across lo- cal context (e.g. sentences, paragraphs, documents), and we capture this variance by adding a component to the model for content states that is based on la- tent Dirichlet allocation (Blei et al., 2003). This ex- tension is in some ways …

Content truncated to 500 words