A fully Bayesian approach to unsupervised part-of-speech tagging

Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pages 744–751, Prague, Czech Republic, June 2007. c⃝2007 Association for Computational Linguistics A Fully Bayesian Approach to Unsupervised Part-of-Speech Tagging∗ Sharon Goldwater Department of Linguistics Stanford University sgwater@stanford.edu Thomas L. Griffiths Department of Psychology UC Berkeley tom griffiths@berkeley.edu Abstract Unsupervised learning of linguistic structure is a difficult problem. A common approach is to define a generative model and max- imize the probability of the hidden struc- ture given the observed data. Typically, this is done using maximum-likelihood es- timation (MLE) of the model parameters. We show using part-of-speech tagging that a fully Bayesian approach can greatly im- prove performance. Rather than estimating a single set of parameters, the Bayesian ap- proach integrates over all possible parame- ter values. This difference ensures that the learned structure will have high probability over a range of possible parameters, and per- mits the use of priors favoring the sparse distributions that are typical of natural lan- guage. Our model has the structure of a standard trigram HMM, yet its accuracy is closer to that of a state-of-the-art discrimi- native model (Smith and Eisner, 2005), up to 14 percentage points better than MLE. We find improvements both when training from data alone, and using a tagging dictionary. 1 Introduction Unsupervised learning of linguistic structure is a dif- ficult problem. Recently, several new model-based approaches have improved performance on a vari- ety of tasks (Klein and Manning, 2002; Smith and ∗This work was supported by grants NSF 0631518 and ONR MURI N000140510388. We would also like to thank Noah Smith for providing us with his data sets. Eisner, 2005). Nearly all of these approaches have one aspect in common: the goal of learning is to identify the set of model parameters that maximizes some objective function. Values for the hidden vari- ables in the model are then chosen based on the learned parameterization. Here, we propose a dif- ferent approach based on Bayesian statistical prin- ciples: rather than searching for an optimal set of parameter values, we seek to directly maximize the probability of the hidden variables given the ob- served data, integrating over all possible parame- ter values. Using part-of-speech (POS) tagging as an example application, we show that the Bayesian approach provides large performance improvements over maximum-likelihood estimation (MLE) for the same model structure. Two factors can explain the improvement. First, integrating over parameter val- ues leads to greater robustness in the choice of tag sequence, since it must have high probability over a range of parameters. Second, integration permits the use of priors favoring sparse distributions, which are typical of natural language. These kinds of pri- ors can lead to degenerate solutions if the parameters are estimated directly. Before describing our approach in more detail, we briefly review previous work on unsupervised POS tagging. Perhaps the most well-known is that of Merialdo (1994), who used MLE to train a tri- gram hidden Markov model (HMM). More recent work has shown that …

Content truncated to 500 words