INESC-ID at SemEval-2016 Task 4-A: Reducing the Problem of Out-of-Embedding Words

Proceedings of SemEval-2016, pages 238–242, San Diego, California, June 16-17, 2016. c⃝2016 Association for Computational Linguistics INESC-ID at SemEval-2016 Task 4-A: Reducing the Problem of Out-of-Embedding Words Silvio Amir, Ramon F. Astudillo, Wang Ling, M´ario J. Silva, Isabel Trancoso Instituto de Engenharia de Sistemas e Computadores Investigac¸˜ao e Desenvolvimento Rua Alves Redol 9 Lisbon, Portugal {samir,ramon.astudillo,wlin,mjs,isabel.trancoso}@inesc-id.pt Abstract We present the INESC-ID system for the 2016 edition of SemEval Twitter Sentiment Analy- sis shared task (subtask 4-A). The system was based on the Non-Linear Sub-space Embed- ding (NLSE) model developed for last year’s competition. This model trains a projection of pre-trained embeddings into a small sub- space using the supervised data available. De- spite its simplicity, the system attained perfor- mances comparable to the best systems of last edition with no need for feature engineering. One limitation of this model was the assump- tion that a pre-trained embedding was avail- able for every word. In this paper, we investi- gated different strategies to overcome this lim- itation by exploiting character-level embed- dings and learning representations for out-of- embedding vocabulary words. The resulting approach outperforms our previous model by a relatively small margin, while still attaining strong results and a consistent good perfor- mance across all the evaluation datasets. 1 Introduction Pre-trained word embeddings provide a simple means to attain semi-supervised learning in Natu- ral Language Processing (NLP) tasks (Collobert et al., 2011). They can be trained with large amounts of unsupervised data and be fine tuned as the initial building block of a semi-supervised system. How- ever, in domains with a significant number of typos, use of slang and abbreviations, such as social me- dia, the high number of singletons leads to a poor fine tuning of the embeddings. In previous work, we addressed this by learning a projection of the embeddings into a small sub-space (Astudillo et al., 2015b). This allowed us to attain representa- tions also for Out-Of-Vocabulary (OOV) words, pro- vided that embeddings for those words are avail- able. However, even if the embeddings are esti- mated from large amounts of unlabeled text, in noisy domains, such as Twitter, a significant number of words will not be seen and therefore will not have an embedding. We refer to those words as the Out- of-Embedding Vocabulary (OOEV). In this paper, we focus on the problem of obtain- ing good representations for OOEV words. We ex- perimented with character to word models (C2W) and investigated different strategies for initializing and updating OOEVs from the available training data. The best results were attained by using the la- beled data to perform small updates to these repre- sentations in the first few epochs of training. The re- sulting system outperforms that of the previous eval- uation, although by a small margin. It ranks fourth in the 2016 evaluation with a consistently high per- formance in all years. 2 NLSE Model Overview In this section, we briefly review the approach in- troduced in the 2015 evaluation (Astudillo et al., 2015a). For a particular …

Content truncated to 500 words