Weakly-Supervised Learning with Cost-Augmented Contrastive Estimation

Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1329–1341, October 25-29, 2014, Doha, Qatar. c⃝2014 Association for Computational Linguistics Weakly-Supervised Learning with Cost-Augmented Contrastive Estimation Kevin Gimpel Mohit Bansal Toyota Technological Institute at Chicago, IL 60637, USA {kgimpel,mbansal}@ttic.edu Abstract We generalize contrastive estimation in two ways that permit adding more knowl- edge to unsupervised learning. The first allows the modeler to specify not only the set of corrupted inputs for each observa- tion, but also how bad each one is. The second allows specifying structural prefer- ences on the latent variable used to explain the observations. They require setting ad- ditional hyperparameters, which can be problematic in unsupervised learning, so we investigate new methods for unsuper- vised model selection and system com- bination. We instantiate these ideas for part-of-speech induction without tag dic- tionaries, improving over contrastive esti- mation as well as strong benchmarks from the PASCAL 2012 shared task. 1 Introduction Unsupervised NLP aims to discover useful struc- ture in unannotated text. This structure might be part-of-speech (POS) tag sequences (Merialdo, 1994), morphological segmentation (Creutz and Lagus, 2005), or syntactic structure (Klein and Manning, 2004), among others. Unsupervised systems typically improve when researchers incor- porate knowledge to bias learning to capture char- acteristics of the desired structure.1 There are many successful examples of adding knowledge to improve learning without labeled examples, including: sparsity in POS tag distri- butions (Johnson, 2007; Ravi and Knight, 2009; Ganchev et al., 2010), short attachments for dependency parsing (Smith and Eisner, 2006), 1We note that doing so strains the definition of the term unsupervised. Hence we will use the term weakly-supervised to refer to methods that do not explicitly train on labeled ex- amples for the task of interest, but do use some form of task- specific knowledge. agreement of word alignment models (Liang et al., 2006), power law effects in lexical distribu- tions (Blunsom and Cohn, 2010; Blunsom and Cohn, 2011), multilingual constraints (Smith and Eisner, 2009; Ganchev et al., 2009; Snyder et al., 2009; Das and Petrov, 2011), and orthographic cues (Spitkovsky et al., 2010c; Spitkovsky et al., 2011b), inter alia. Contrastive estimation (CE; Smith and Eisner, 2005) is a general approach to weakly-supervised learning with a particular way of incorporating knowledge. CE increases the likelihood of the ob- servations at the expense of those in a particular neighborhood of each observation. The neighbor- hood typically contains corrupted versions of the observations. The latent structure is marginalized out for both the observations and their corruptions; the intent is to learn latent structure that helps to explain why the observation was generated rather than any of the corrupted alternatives. In this paper, we present a new objective func- tion for weakly-supervised learning that general- izes CE by including two types of cost functions, one on observations and one on output structures. The first (§4) allows us to specify not only the set of corrupted observations, but also how bad each corruption was. We use n-gram language models …

Content truncated to 500 words