A comparison of Bayesian estimators for unsupervised Hidden Markov Model POS taggers

Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pages 344–352, Honolulu, October 2008. c⃝2008 Association for Computational Linguistics A comparison of Bayesian estimators for unsupervised Hidden Markov Model POS taggers Jianfeng Gao Microsoft Research Redmond, WA, USA jfgao@microsoft.com Mark Johnson Brown Univeristy Providence, RI, USA Mark−Johnson@Brown.edu Abstract There is growing interest in applying Bayesian techniques to NLP problems. There are a number of different estimators for Bayesian models, and it is useful to know what kinds of tasks each does well on. This paper compares a variety of different Bayesian estimators for Hidden Markov Model POS taggers with var- ious numbers of hidden states on data sets of different sizes. Recent papers have given con- tradictory results when comparing Bayesian estimators to Expectation Maximization (EM) for unsupervised HMM POS tagging, and we show that the difference in reported results is largely due to differences in the size of the training data and the number of states in the HMM. We invesigate a variety of samplers for HMMs, including some that these earlier pa- pers did not study. We find that all of Gibbs samplers do well with small data sets and few states, and that Variational Bayes does well on large data sets and is competitive with the Gibbs samplers. In terms of times of conver- gence, we find that Variational Bayes was the fastest of all the estimators, especially on large data sets, and that explicit Gibbs sampler (both pointwise and sentence-blocked) were gener- ally faster than their collapsed counterparts on large data sets. 1 Introduction Probabilistic models now play a central role in com- putational linguistics. These models define a prob- ability distribution P(x) over structures or analyses x. For example, in the part-of-speech (POS) tag- ging application described in this paper, which in- volves predicting the part-of-speech tag ti of each word wi in the sentence w = (w1, . . . , wn), the structure x = (w, t) consists of the words w in a sentence together with their corresponding parts-of- speech t = (t1, . . . , tn). In general the probabilistic models used in com- putational linguistics have adjustable parameters θ which determine the distribution P(x | θ). In this paper we focus on bitag Hidden Markov Models (HMMs). Since our goal here is to compare algo- rithms rather than achieve the best performance, we keep the models simple by ignoring morphology and capitalization (two very strong cues in English) and treat each word as an atomic entity. This means that the model parameters θ consist of the HMM state- to-state transition probabilities and the state-to-word emission probabilities. In virtually all statistical approaches the parame- ters θ are chosen or estimated on the basis of training data d. This paper studies unsupervised estimation, so d = w = (w1, . . . , wn) consists of a sequence of words wi containing all of the words of training corpus appended into a single string, as explained below. Maximum …

Content truncated to 500 words