Language Models as Representations for Weakly Supervised NLP Tasks

Proceedings of the Fifteenth Conference on Computational Natural Language Learning, pages 125–134, Portland, Oregon, USA, 23–24 June 2011. c⃝2011 Association for Computational Linguistics Language Models as Representations for Weakly-Supervised NLP Tasks Fei Huang and Alexander Yates Temple University Broad St. and Montgomery Ave. Philadelphia, PA 19122 fei.huang@temple.edu yates@temple.edu Arun Ahuja and Doug Downey Northwestern University 2133 Sheridan Road Evanston, IL 60208 a-ahuja@northwestern.edu ddowney@eecs.northwestern.edu Abstract Finding the right representation for words is critical for building accurate NLP systems when domain-specific labeled data for the task is scarce. This paper investigates lan- guage model representations, in which lan- guage models trained on unlabeled corpora are used to generate real-valued feature vec- tors for words. We investigate ngram mod- els and probabilistic graphical models, includ- ing a novel lattice-structured Markov Random Field. Experiments indicate that language model representations outperform traditional representations, and that graphical model rep- resentations outperform ngram models, espe- cially on sparse and polysemous words. 1 Introduction NLP systems often rely on hand-crafted, carefully engineered sets of features to achieve strong perfor- mance. Thus, a part-of-speech (POS) tagger would traditionally use a feature like, “the previous token is the” to help classify a given token as a noun or adjective. For supervised NLP tasks with sufficient domain-specific training data, these traditional fea- tures yield state-of-the-art results. However, NLP systems are increasingly being applied to texts like the Web, scientific domains, and personal commu- nications like emails, all of which have very differ- ent characteristics from traditional training corpora. Collecting labeled training data for each new target domain is typically prohibitively expensive. We in- vestigate representations that can be applied when domain-specific labeled training data is scarce. An increasing body of theoretical and empirical evidence suggests that traditional, manually-crafted features limit systems’ performance in this setting for two reasons. First, feature sparsity prevents sys- tems from generalizing accurately to words and fea- tures not seen during training. Because word fre- quencies are Zipf distributed, this often means that there is little relevant training data for a substantial fraction of parameters (Bikel, 2004), especially in new domains (Huang and Yates, 2009). For exam- ple, word-type features form the backbone of most POS-tagging systems, but types like “gene” and “pathway” show up frequently in biomedical liter- ature, and rarely in newswire text. Thus, a classifier trained on newswire data and tested on biomedical data will have seen few training examples related to sentences with features “gene” and “pathway” (Ben- David et al., 2009; Blitzer et al., 2006). Further, because words are polysemous, word- type features prevent systems from generalizing to situations in which words have different meanings. For instance, the word type “signaling” appears pri- marily as a present participle (VBG) in Wall Street Journal (WSJ) text, as in, “Interest rates rose, sig- naling that . . . ” (Marcus et al., 1993). In biomedical text, however, “signaling” appears primarily in the phrase “signaling pathway,” where it is considered a noun (NN) (PennBioIE, 2005); this phrase never appears in the WSJ portion of the …

Content truncated to 500 words