Learning Word Representations from Scarce and Noisy Data with Embedding Subspaces

Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, pages 1074–1084, Beijing, China, July 26-31, 2015. c⃝2015 Association for Computational Linguistics Learning Word Representations from Scarce and Noisy Data with Embedding Sub-spaces Ramon F. Astudillo, Silvio Amir, Wang Lin, M´ario Silva, Isabel Trancoso Instituto de Engenharia de Sistemas e Computadores Investigac¸˜ao e Desenvolvimento (INESC-ID) Rua Alves Redol 9 Lisbon, Portugal {ramon.astudillo, samir, wlin, mjs, isabel.trancoso}@inesc-id.pt Abstract We investigate a technique to adapt unsu- pervised word embeddings to specific ap- plications, when only small and noisy la- beled datasets are available. Current meth- ods use pre-trained embeddings to initial- ize model parameters, and then use the la- beled data to tailor them for the intended task. However, this approach is prone to overfitting when the training is performed with scarce and noisy data. To overcome this issue, we use the supervised data to find an embedding subspace that fits the task complexity. All the word representa- tions are adapted through a projection into this task-specific subspace, even if they do not occur on the labeled dataset. This ap- proach was recently used in the SemEval 2015 Twitter sentiment analysis challenge, attaining state-of-the-art results. Here we show results improving those of the chal- lenge, as well as additional experiments in a Twitter Part-Of-Speech tagging task. 1 Introduction The success of supervised systems largely depends on the amount and quality of the available train- ing data, oftentimes, even more than the particu- lar choice of learning algorithm (Banko and Brill, 2001). Labeled data is, however, expensive to ob- tain, while unlabeled data is widely available. In order to exploit this fact, semi-supervised learn- ing methods can be used. In particular, it is pos- sible to derive word representations by exploiting word co-occurrence patterns in large samples of unlabeled text. Based on this idea, several meth- ods have been recently proposed to efficiently es- timate word embeddings from raw text, leverag- ing neural language models (Huang et al., 2012; Mikolov et al., 2013; Pennington et al., 2014; Ling et al., 2015). These models work by maximizing the probability that words within a given window size are predicted correctly. The resulting embed- dings are low-dimensional dense vectors that en- code syntactic and semantic properties of words. Using these word representations, Turian et al. (2010) were able to improve near state-of-the-art systems for several tasks, by simply plugging in the learned word representations as additional fea- tures. However, because these features are esti- mated by minimizing the prediction errors made on a generic, unsupervised, task they might be suboptimal for the intended purposes. Ideally, word features should be adapted to the specific supervised task. One of the reasons for the success of deep learning models for language problems, is the use unsupervised word embed- dings to initialize the word projection layer. Then, during training, the errors made in the predictions are backpropagated to update the embeddings, so that they better predict the supervised …

Content truncated to 500 words