Simple Type-Level Unsupervised POS Tagging

Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pages 853–861, MIT, Massachusetts, USA, 9-11 October 2010. c⃝2010 Association for Computational Linguistics Simple Type-Level Unsupervised POS Tagging Yoong Keok Lee Aria Haghighi Regina Barzilay Computer Science and Artificial Intelligence Laboratory Massachusetts Institute of Technology {yklee, aria42, regina}@csail.mit.edu Abstract Part-of-speech (POS) tag distributions are known to exhibit sparsity — a word is likely to take a single predominant tag in a corpus. Recent research has demonstrated that incor- porating this sparsity constraint improves tag- ging accuracy. However, in existing systems, this expansion come with a steep increase in model complexity. This paper proposes a sim- ple and effective tagging method that directly models tag sparsity and other distributional properties of valid POS tag assignments. In addition, this formulation results in a dramatic reduction in the number of model parame- ters thereby, enabling unusually rapid training. Our experiments consistently demonstrate that this model architecture yields substantial per- formance gains over more complex tagging counterparts. On several languages, we report performance exceeding that of more complex state-of-the art systems.1 1 Introduction Since the early days of statistical NLP, researchers have observed that a part-of-speech tag distribution exhibits “one tag per discourse” sparsity — words are likely to select a single predominant tag in a cor- pus, even when several tags are possible. Simply assigning to each word its most frequent associated tag in a corpus achieves 94.6% accuracy on the WSJ portion of the Penn Treebank. This distributional sparsity of syntactic tags is not unique to English 1The source code for the work presented in this paper is available at http://groups.csail.mit.edu/rbg/code/typetagging/. — similar results have been observed across multi- ple languages. Clearly, explicitly modeling such a powerful constraint on tagging assignment has a po- tential to significantly improve the accuracy of an unsupervised part-of-speech tagger learned without a tagging dictionary. In practice, this sparsity constraint is difficult to incorporate in a traditional POS induction sys- tem (M´erialdo, 1994; Johnson, 2007; Gao and John- son, 2008; Grac¸a et al., 2009; Berg-Kirkpatrick et al., 2010). These sequence models-based ap- proaches commonly treat token-level tag assignment as the primary latent variable. By design, they read- ily capture regularities at the token-level. However, these approaches are ill-equipped to directly repre- sent type-based constraints such as sparsity. Pre- vious work has attempted to incorporate such con- straints into token-level models via heavy-handed modifications to inference procedure and objective function (e.g., posterior regularization and ILP de- coding) (Grac¸a et al., 2009; Ravi and Knight, 2009). In most cases, however, these expansions come with a steep increase in model complexity, with respect to training procedure and inference time. In this work, we take a more direct approach and treat a word type and its allowed POS tags as a pri- mary element of the model. The model starts by gen- erating a tag assignment for each word type in a vo- cabulary, assuming one tag per word. Then, token- level HMM emission parameters are drawn condi- tioned …

Content truncated to 500 words