Compound Embedding Features for Semi-supervised Learning

Proceedings of NAACL-HLT 2013, pages 563–568, Atlanta, Georgia, 9–14 June 2013. c⃝2013 Association for Computational Linguistics Compound Embedding Features for Semi-supervised Learning Mo Yu1, Tiejun Zhao1, Daxiang Dong2, Hao Tian2 and Dianhai Yu2 Harbin Institute of Technology, Harbin, China Baidu Inc., Beijing, China {yumo,tjzhao}@mtlab.hit.edu.cn {dongdaxiang,tianhao,yudianhai}@baidu.com Abstract To solve data sparsity problem, recently there has been a trend in discriminative methods of NLP to use representations of lexical items learned from unlabeled data as features. In this paper, we investigated the usage of word representations learned by neural language models, i.e. word embeddings. The direct us- age has disadvantages such as large amount of computation, inadequacy with dealing word ambiguity and rare-words, and the problem of linear non-separability. To overcome these problems, we instead built compound features from continuous word embeddings based on clustering. Experiments showed that the com- pound features not only improved the perfor- mances on several NLP tasks, but also ran faster, suggesting the potential of embeddings. 1 Introduction Supervised learning methods have achieved great successes in the field of Natural Language Pro- cessing (NLP). However, in practice most methods are usually limited by the problem of data sparsity, since it is impossible to obtain sufficient labeled data for all NLP tasks. In these situations semi- supervised learning can help to make use of both labeled data and easy-to-obtain unlabeled data. The semi-supervised framework that is widely applied to NLP is to first learn word representa- tions, which are feature vectors of lexical items, from unlabeled data and then plug them into a su- pervised system. These methods are very effective in utilizing large-scale unlabeled data and have successfully improved performances of state-of- the-art supervised systems on a variety of tasks (Koo et al., 2008; Huang and Yates, 2009; Täck- ström et al., 2012). With the development of neural language mod- els (NLM) (Bengio et al., 2003; Mnih and Hinton, 2009), recently researchers become interested in word representations (also called word embed- dings) learned by these models. Word embeddings are dense low dimensional real-valued vectors. They are composed of some latent features, which are expected to capture useful syntactic and seman- tic properties. Word embeddings are usually served as the first layer in deep learning systems for NLP (Collobert and Weston, 2008; Socher et al., 2011a, 2011b) and help these systems perform compara- bly with the state-of-the-art models based on hand- crafted features. They also have been directly added as features to the state-of-the-art models of chunking and NER, and have achieved significant improvements (Turian et al. 2010). Although the direct usage of continuous embed- dings has been proved to be an effective method for enhancing the state-of-the-art supervised mod- els, it has some disadvantages, which made them be out-performed by simpler Brown cluster fea- tures (Turian et al, 2010) and made them computa- tionally complicated. Firstly, embeddings of rare words are insufficiently trained since they are only updated few times and are close to their random initial values. As shown in (Turian et al, 2010), this is …

Content truncated to 500 words