Revisiting Embedding Features for Simple Semi-supervised Learning

Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 110–120, October 25-29, 2014, Doha, Qatar. c⃝2014 Association for Computational Linguistics Revisiting Embedding Features for Simple Semi-supervised Learning Jiang Guo†, Wanxiang Che†, Haifeng Wang‡, Ting Liu†∗ †Research Center for Social Computing and Information Retrieval Harbin Institute of Technology, China ‡Baidu Inc., Beijing, China {jguo, car, tliu}@ir.hit.edu.cn wanghaifeng@baidu.com Abstract Recent work has shown success in us- ing continuous word embeddings learned from unlabeled data as features to improve supervised NLP systems, which is re- garded as a simple semi-supervised learn- ing mechanism. However, fundamen- tal problems on effectively incorporating the word embedding features within the framework of linear models remain. In this study, we investigate and analyze three different approaches, including a new pro- posed distributional prototype approach, for utilizing the embedding features. The presented approaches can be integrated into most of the classical linear models in NLP. Experiments on the task of named entity recognition show that each of the proposed approaches can better utilize the word embedding features, among which the distributional prototype approach per- forms the best. Moreover, the combination of the approaches provides additive im- provements, outperforming the dense and continuous embedding features by nearly 2 points of F1 score. 1 Introduction Learning generalized representation of words is an effective way of handling data sparsity caused by high-dimensional lexical features in NLP sys- tems, such as named entity recognition (NER) and dependency parsing. As a typical low- dimensional and generalized word representa- tion, Brown clustering of words has been stud- ied for a long time. For example, Liang (2005) and Koo et al. (2008) used the Brown cluster features for semi-supervised learning of various NLP tasks and achieved significant improvements. ∗Email correspondence. Recent research has focused on a special fam- ily of word representations, named “word embed- dings”. Word embeddings are conventionally de- fined as dense, continuous, and low-dimensional vector representations of words. Word embed- dings can be learned from large-scale unlabeled texts through context-predicting models (e.g., neu- ral network language models) or spectral methods (e.g., canonical correlation analysis) in an unsu- pervised setting. Compared with the so-called one-hot represen- tation where each word is represented as a sparse vector of the same size of the vocabulary and only one dimension is on, word embedding preserves rich linguistic regularities of words with each di- mension hopefully representing a latent feature. Similar words are expected to be distributed close to one another in the embedding space. Conse- quently, word embeddings can be beneficial for a variety of NLP applications in different ways, among which the most simple and general way is to be fed as features to enhance existing supervised NLP systems. Previous work has demonstrated effectiveness of the continuous word embedding features in sev- eral tasks such as chunking and NER using gener- alized linear models (Turian et al., 2010).1 How- ever, there still remain two fundamental problems that should be addressed: • Are the continuous embedding features fit for the generalized linear …

Content truncated to 500 words