Word Representations: A Simple and General Method for Semi-Supervised Learning

Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 384–394, Uppsala, Sweden, 11-16 July 2010. c⃝2010 Association for Computational Linguistics Word representations: A simple and general method for semi-supervised learning Joseph Turian D´epartement d’Informatique et Recherche Op´erationnelle (DIRO) Universit´e de Montr´eal Montr´eal, Qu´ebec, Canada, H3T 1J4 lastname@iro.umontreal.ca Lev Ratinov Department of Computer Science University of Illinois at Urbana-Champaign Urbana, IL 61801 ratinov2@uiuc.edu Yoshua Bengio D´epartement d’Informatique et Recherche Op´erationnelle (DIRO) Universit´e de Montr´eal Montr´eal, Qu´ebec, Canada, H3T 1J4 bengioy@iro.umontreal.ca Abstract If we take an existing supervised NLP sys- tem, a simple and general way to improve accuracy is to use unsupervised word representations as extra word features. We evaluate Brown clusters, Collobert and Weston (2008) embeddings, and HLBL (Mnih & Hinton, 2009) embeddings of words on both NER and chunking. We use near state-of-the-art supervised baselines, and find that each of the three word representations improves the accu- racy of these baselines. We find further improvements by combining different word representations. You can download our word features, for off-the-shelf use in existing NLP systems, as well as our code, here: http://metaoptimize. com/projects/wordreprs/ 1 Introduction By using unlabelled data to reduce data sparsity in the labeled training data, semi-supervised approaches improve generalization accuracy. Semi-supervised models such as Ando and Zhang (2005), Suzuki and Isozaki (2008), and Suzuki et al. (2009) achieve state-of-the-art accuracy. However, these approaches dictate a particular choice of model and training regime. It can be tricky and time-consuming to adapt an existing su- pervised NLP system to use these semi-supervised techniques. It is preferable to use a simple and general method to adapt existing supervised NLP systems to be semi-supervised. One approach that is becoming popular is to use unsupervised methods to induce word features—or to download word features that have already been induced—plug these word features into an existing system, and observe a significant increase in accuracy. But which word features are good for what tasks? Should we prefer certain word features? Can we combine them? A word representation is a mathematical object associated with each word, often a vector. Each dimension’s value corresponds to a feature and might even have a semantic or grammatical interpretation, so we call it a word feature. Conventionally, supervised lexicalized NLP ap- proaches take a word and convert it to a symbolic ID, which is then transformed into a feature vector using a one-hot representation: The feature vector has the same length as the size of the vocabulary, and only one dimension is on. However, the one-hot representation of a word suffers from data sparsity: Namely, for words that are rare in the labeled training data, their corresponding model parameters will be poorly estimated. Moreover, at test time, the model cannot handle words that do not appear in the labeled training data. These limitations of one-hot word representations have prompted researchers to investigate unsupervised methods for inducing word representations over large unlabeled corpora. Word features can be hand-designed, but our goal is to learn them. One common …

Content truncated to 500 words