Painless Unsupervised Learning with Features

Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the ACL, pages 582–590, Los Angeles, California, June 2010. c⃝2010 Association for Computational Linguistics Painless Unsupervised Learning with Features Taylor Berg-Kirkpatrick Alexandre Bouchard-Cˆot´e John DeNero Dan Klein Computer Science Division University of California at Berkeley {tberg, bouchard, denero, klein}@cs.berkeley.edu Abstract We show how features can easily be added to standard generative models for unsuper- vised learning, without requiring complex new training methods. In particular, each component multinomial of a generative model can be turned into a miniature logistic regres- sion model if feature locality permits. The in- tuitive EM algorithm still applies, but with a gradient-based M-step familiar from discrim- inative training of logistic regression mod- els. We apply this technique to part-of-speech induction, grammar induction, word align- ment, and word segmentation, incorporating a few linguistically-motivated features into the standard generative model for each task. These feature-enhanced models each outper- form their basic counterparts by a substantial margin, and even compete with and surpass more complex state-of-the-art models. 1 Introduction Unsupervised learning methods have been increas- ingly successful in recent NLP research. The rea- sons are varied: increased supplies of unlabeled data, improved understanding of modeling methods, additional choices of optimization algorithms, and, perhaps most importantly for the present work, in- corporation of richer domain knowledge into struc- tured models. Unfortunately, that knowledge has generally been encoded in the form of conditional independence structure, which means that injecting it is both tricky (because the connection between independence and knowledge is subtle) and time- consuming (because new structure often necessitates new inference algorithms). In this paper, we present a range of experiments wherein we improve existing unsupervised models by declaratively adding richer features. In particu- lar, we parameterize the local multinomials of exist- ing generative models using features, in a way which does not require complex new machinery but which still provides substantial flexibility. In the feature- engineering paradigm, one can worry less about the backbone structure and instead use hand-designed features to declaratively inject domain knowledge into a model. While feature engineering has his- torically been associated with discriminative, super- vised learning settings, we argue that it can and should be applied more broadly to the unsupervised setting. The idea of using features in unsupervised learn- ing is neither new nor even controversial. Many top unsupervised results use feature-based mod- els (Smith and Eisner, 2005; Haghighi and Klein, 2006). However, such approaches have presented their own barriers, from challenging normalization problems, to neighborhood design, to the need for complex optimization procedures. As a result, most work still focuses on the stable and intuitive ap- proach of using the EM algorithm to optimize data likelihood in locally normalized, generative models. The primary contribution of this paper is to demonstrate the clear empirical success of a sim- ple and accessible approach to unsupervised learn- ing with features, which can be optimized by us- ing standard NLP building blocks. We consider the same generative, locally-normalized models that dominate past work …

Content truncated to 500 words