The infinite HMM for unsupervised PoS tagging

Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, pages 678–687, Singapore, 6-7 August 2009. c⃝2009 ACL and AFNLP The infinite HMM for unsupervised PoS tagging Jurgen Van Gael Department of Engineering University of Cambridge jv249@cam.ac.uk Andreas Vlachos Computer Laboratory University of Cambridge av308@cl.cam.ac.uk Zoubin Ghahramani Department of Engineering University of Cambridge zoubin@eng.cam.ac.uk Abstract We extend previous work on fully unsu- pervised part-of-speech tagging. Using a non-parametric version of the HMM, called the infinite HMM (iHMM), we ad- dress the problem of choosing the number of hidden states in unsupervised Markov models for PoS tagging. We experi- ment with two non-parametric priors, the Dirichlet and Pitman-Yor processes, on the Wall Street Journal dataset using a paral- lelized implementation of an iHMM in- ference algorithm. We evaluate the re- sults with a variety of clustering evalua- tion metrics and achieve equivalent or bet- ter performances than previously reported. Building on this promising result we eval- uate the output of the unsupervised PoS tagger as a direct replacement for the out- put of a fully supervised PoS tagger for the task of shallow parsing and compare the two evaluations. 1 Introduction Many Natural Language Processing (NLP) tasks are commonly tackled using supervised learning approaches. These learning methods rely on the availability of labeled datasets which are usually produced by expensive manual annotation. For some tasks, we have the choice to use unsuper- vised learning approaches. While they do not nec- essarily achieve the same level of performance, they are appealing as unlabeled data is usually abundant. In particular, for the purpose of ex- ploring new domains and languages, obtainining labeled material can be prohibitively expensive and unsupervised learning methods are a very at- tractive choice. Recent work (Johnson, 2007; Goldwater and Griffiths, 2007; Gao and Johnson, 2008) explored the task of part-of-speech tagging (PoS) using unsupervised Hidden Markov Models (HMMs) with encouraging results. PoS tagging is a standard component in many linguistic process- ing pipelines, so any improvement on its perfor- mance is likely to impact a wide range of tasks. It is important to point out that a completely unsupervised learning method will discover the statistics of a dataset according to a particular model choice but these statistics might not cor- respond exactly to our intuition about PoS tags. Johnson (2007) and Gao & Johnson (2008) as- sume that words are generated by a hidden Markov model and find that the resulting states strongly correlate with POS tags. Nonetheless, identifying the HMM states with appropriate POS tags is hard. Because many evaluation methods often require POS tags (rather than HMM states) this identifica- tion problem makes unsupervised systems difficult to evaluate. One potential solution is to add a small amount of supervision as in Goldwater & Griffiths (2007) who assume a dictionary of frequent words asso- ciated with possible PoS tags extracted from a la- beled corpus. Although this technique improves performance, in this paper we explore the com- pletely unsupervised approach. The reason for this is that …

Content truncated to 500 words