Unsupervised POS Induction with Word Embeddings

Human Language Technologies: The 2015 Annual Conference of the North American Chapter of the ACL, pages 1311–1316, Denver, Colorado, May 31 – June 5, 2015. c⃝2015 Association for Computational Linguistics Unsupervised POS Induction with Word Embeddings Chu-Cheng Lin Waleed Ammar Chris Dyer Lori Levin School of Computer Science Carnegie Mellon University {chuchenl,wammar,cdyer,lsl}@cs.cmu.edu Abstract Unsupervised word embeddings have been shown to be valuable as features in supervised learning problems; however, their role in unsu- pervised problems has been less thoroughly ex- plored. In this paper, we show that embeddings can likewise add value to the problem of unsu- pervised POS induction. In two representative models of POS induction, we replace multi- nomial distributions over the vocabulary with multivariate Gaussian distributions over word embeddings and observe consistent improve- ments in eight languages. We also analyze the effect of various choices while inducing word embeddings on “downstream” POS induction results. 1 Introduction Unsupervised POS induction is the problem of as- signing word tokens to syntactic categories given only a corpus of untagged text. In this paper we ex- plore the effect of replacing words with their vector space embeddings1 in two POS induction models: the classic first-order HMM (Kupiec, 1992) and the newly introduced conditional random field autoen- coder (Ammar et al., 2014). In each model, instead of using a conditional multinomial distribution2 to generate a word token wi ∈ V given a POS tag ti ∈ T, we use a conditional Gaussian distribution and gen- erate a d-dimensional word embedding vwi ∈ Rd given ti. 1Unlike Yatbaz et al. (2014), we leverage easily obtainable and widely used embeddings of word types. 2Also known as a categorical distribution. Our findings suggest that, in both models, sub- stantial improvements are possible when word em- beddings are used rather than opaque word types. However, the independence assumptions made by the model used to induce embeddings strongly deter- mines its effectiveness for POS induction: embedding models that model short-range context are more ef- fective than those that model longer-range contexts. This result is unsurprising, but it illustrates the lack of an evaluation metric that measures the syntactic (rather than semantic) information in word embed- dings. Our results also confirm the conclusions of Sirts et al. (2014) who were likewise able to improve POS induction results, albeit using a custom clus- tering model based on the the distance-dependent Chinese restaurant process (Blei and Frazier, 2011). Our contributions are as follows: (i) reparameter- ization of token-level POS induction models to use word embeddings; and (ii) a systematic evaluation of word embeddings with respect to the syntactic information they contain. 2 Vector Space Word Embeddings Word embeddings represent words in a language’s vocabulary as points in a d-dimensional space such that nearby words (points) are similar in terms of their distributional properties. A variety of techniques for learning embeddings have been proposed, e.g., matrix factorization (Deerwester et al., 1990; Dhillon et al., 2011) and neural language modeling (Mikolov et al., 2011; Collobert and Weston, 2008). For the POS induction …

Content truncated to 500 words