Not All Contexts Are Created Equal: Better Word Representations with Variable Attention

Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1367–1372, Lisbon, Portugal, 17-21 September 2015. c⃝2015 Association for Computational Linguistics. Not All Contexts Are Created Equal: Better Word Representations with Variable Attention Wang Ling Lin Chu-Cheng Yulia Tsvetkov Silvio Amir Ram´on Fernandez Astudillo Chris Dyer Alan W Black Isabel Trancoso L2F Spoken Systems Lab, INESC-ID, Lisbon, Portugal Language Technologies Institute, Carnegie Mellon University, Pittsburgh, PA, USA Instituto Superior T´ecnico, Lisbon, Portugal {lingwang,chuchenl,ytsvetko,cdyer,awb}@cs.cmu.edu {ramon.astudillo,samir,isabel.trancoso}@inesc-id.pt Abstract We introduce an extension to the bag-of- words model for learning words represen- tations that take into account both syn- tactic and semantic properties within lan- guage. This is done by employing an at- tention model that finds within the con- textual words, the words that are relevant for each prediction. The general intuition of our model is that some words are only relevant for predicting local context (e.g. function words), while other words are more suited for determining global con- text, such as the topic of the document. Experiments performed on both semanti- cally and syntactically oriented tasks show gains using our model over the existing bag of words model. Furthermore, com- pared to other more sophisticated models, our model scales better as we increase the size of the context of the model. 1 Introduction Learning word representations using raw text data have been shown to improve many NLP tasks, such as part-of-speech tagging (Collobert et al., 2011), dependency parsing (Chen and Manning, 2014; Kong et al., 2014) and machine transla- tion (Liu et al., 2014; Kalchbrenner and Blunsom, 2013; Devlin et al., 2014; Sutskever et al., 2014). These embeddings are generally learnt by defining an objective function, which predicts words con- ditioned on the context surrounding those words. Once trained, these can be used as features (Turian et al., 2010), as initializations of other neural net- works (Hinton and Salakhutdinov, 2012; Erhan et al., 2010; Guo et al., 2014). The continuous bag-of-words (Mikolov et al., 2013) is one of the many models that learns word representations from raw textual data. While these models are adequate for learning semantic fea- tures, one of the problems of this model is the lack of sensitivity for word order, which limits their ability of learn syntactically motivated em- beddings (Ling et al., 2015a; Bansal et al., 2014). While models have been proposed to address this problem, the complexity of these models (“Struc- tured skip-n-gram” and “CWindow”) grows lin- early as size of the window of words considered increases, as a new set of parameters is created for each relative position. On the other hand, the continuous bag-of-words model requires no addi- tional parameters as it builds the context repre- sentation by summing over the embeddings in the window and its performance is an order of magni- tude higher than of other models. In this work, we propose an extension to the continuous bag-of-words model, which adds an at- tention model that considers contextual words dif- ferently depending on the word type and its rela- …

Content truncated to 500 words