Modelling the Lexicon in Unsupervised Part of Speech Induction

Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, pages 116–125, Gothenburg, Sweden, April 26-30 2014. c⃝2014 Association for Computational Linguistics Modelling the Lexicon in Unsupervised Part of Speech Induction Greg Dubbin Department of Computer Science University of Oxford United Kingdom Gregory.Dubbin@wolfson.ox.ac.uk Phil Blunsom Department of Computer Science University of Oxford United Kingdom Phil.Blunsom@cs.ox.ac.uk Abstract Automatically inducing the syntactic part- of-speech categories for words in text is a fundamental task in Computational Linguistics. While the performance of unsupervised tagging models has been slowly improving, current state-of-the-art systems make the obviously incorrect as- sumption that all tokens of a given word type must share a single part-of-speech tag. This one-tag-per-type heuristic coun- ters the tendency of Hidden Markov Model based taggers to over generate tags for a given word type. However, it is clearly incompatible with basic syntactic theory. In this paper we extend a state-of- the-art Pitman-Yor Hidden Markov Model tagger with an explicit model of the lexi- con. In doing so we are able to incorpo- rate a soft bias towards inducing few tags per type. We develop a particle filter for drawing samples from the posterior of our model and present empirical results that show that our model is competitive with and faster than the state-of-the-art without making any unrealistic restrictions. 1 Introduction Research on the unsupervised induction of part- of-speech (PoS) tags has the potential to im- prove both our understanding of the plausibil- ity of theories of first language acquisition, and Natural Language Processing applications such as Speech Recognition and Machine Transla- tion. While there has been much prior work on this task (Brown et al., 1992; Clark, 2003; Christodoulopoulos et al., 2010; Toutanova and Johnson, 2008; Goldwater and Griffiths, 2007; Blunsom and Cohn, 2011), a common thread in many of these works is that models based on a Hidden Markov Model (HMM) graphical struc- ture suffer from a tendency to assign too many different tags to the tokens of a given word type. Models which restrict word types to only occur with a single tag show a significant increase in performance, even though this restriction is clearly at odds with the gold standard labeling (Brown et al., 1992; Clark, 2003; Blunsom and Cohn, 2011). While the empirically observed expectation for the number of tags per word type is close to one, there are many exceptions, e.g. words that occur as both nouns and verbs (opening, increase, related etc.). In this paper we extend the Pitman-Yor HMM tagger (Blunsom and Cohn, 2011) to explicitly in- clude a model of the lexicon that encodes from which tags a word type may be generated. For each word type we draw an ambiguity class which is the set of tags that it may occur with, captur- ing the fact that words are often ambiguous be- tween certain tags (e.g. Noun and Verb), while rarely between others (e.g. Determiner and Verb). We extend the type based Sequential Monte Carlo (SMC) inference algorithm of Dubbin …

Content truncated to 500 words