Type-Supervised Hidden Markov Models for Part-of-Speech Tagging with Incomplete Tag Dictionaries

Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 821–831, Jeju Island, Korea, 12–14 July 2012. c⃝2012 Association for Computational Linguistics Type-Supervised Hidden Markov Models for Part-of-Speech Tagging with Incomplete Tag Dictionaries Dan Garrette Department of Computer Science The University of Texas at Austin dhg@cs.utexas.edu Jason Baldridge Department of Linguistics The University of Texas at Austin jbaldrid@utexas.edu Abstract Past work on learning part-of-speech taggers from tag dictionaries and raw data has re- ported good results, but the assumptions made about those dictionaries are often unrealistic: due to historical precedents, they assume ac- cess to information about labels in the raw and test sets. Here, we demonstrate ways to learn hidden Markov model taggers from in- complete tag dictionaries. Taking the MIN- GREEDY algorithm (Ravi et al., 2010) as a starting point, we improve it with several intu- itive heuristics. We also define a simple HMM emission initialization that takes advantage of the tag dictionary and raw data to capture both the openness of a given tag and its estimated prevalence in the raw data. Altogether, our augmentations produce improvements to per- formance over the original MIN-GREEDY al- gorithm for both English and Italian data. 1 Introduction Learning accurate part-of-speech (POS) taggers based on plentiful labeled training material is gener- ally considered a solved problem. The best taggers obtain accuracies of over 97% for English newswire text in the Penn Treebank, which can be consid- ered as an upper-bound that matches human perfor- mance on the same task (Manning, 2011). How- ever, as Manning notes, this story changes as soon as one is working with different assumptions and data, including having less training data, different kinds of training data, other languages, and other domains. Such POS tagging work has been plen- tiful and includes efforts to induce POS tags without labels (Christodoulopoulos et al., 2010); learn from POS-tag dictionaries (Ravi et al., 2010), incom- plete dictionaries (Hasan and Ng, 2009) and human- constructed dictionaries (Goldberg et al., 2008); bootstrap taggers for a language based on knowl- edge about other languages (Das and Petrov, 2011), and creating supervised taggers for new, challenging domains such as Twitter (Gimpel et al., 2011). Here, we focus on learning from tag dictionar- ies. This is often characterized as unsupervised or weakly supervised training. We adopt the terminol- ogy type-supervised training to distinguish it from unsupervised training from raw text and supervised training from word tokens labeled with their parts- of-speech. Work on type-supervision goes back to (Merialdo, 1994), who introduced the still standard procedure of using a bigram Hidden Markov Model (HMM) trained via Expectation Maximization. Early research appeared to show that learning from types works nearly as well as learning from tokens, with researchers in the 1990s obtaining ac- curacies up to 96% on English (e.g. Kupiec (1992)). However, the tag dictionaries in these cases were ob- tained from labeled tokens. While replicating earlier experiments, Banko and Moore (2004) discovered that performance was highly dependent …

Content truncated to 500 words