Attention Word Embedding

Proceedings of the 28th International Conference on Computational Linguistics, pages 6894–6902 Barcelona, Spain (Online), December 8-13, 2020 6894 Attention Word Embedding Shashank Sonkar Rice University ss164@rice.edu Andrew E. Waters Rice University aew2@rice.edu Richard G. Baraniuk Rice University richb@rice.edu Abstract Word embedding models learn semantically rich vector representations of words and are widely used to initialize natural processing language (NLP) models. The popular continuous bag-of- words (CBOW) model of word2vec learns a vector embedding by masking a given word in a sentence and then using the other words as a context to predict it. A limitation of CBOW is that it equally weights the context words when making a prediction, which is inefficient, since some words have higher predictive value than others. We tackle this inefficiency by introducing the Attention Word Embedding (AWE) model, which integrates the attention mechanism into the CBOW model. We also propose AWE-S, which incorporates subword information. We demonstrate that AWE and AWE-S outperform the state-of-the-art word embedding models both on a variety of word similarity datasets and when used for initialization of NLP models. 1 Introduction Word embedding models learn vector representations of words such that words that are semantically related are close to each other in the vector space. Popular word embedding models include word2vec (Mikolov et al., 2013a; Mikolov et al., 2013b), GloVe (Pennington et al., 2014), and fastText (Bojanowski et al., 2017). Word embedding models are often used to initialize deep learning models for various natural language processing (NLP) tasks such as machine translation (Bahdanau et al., 2015), part-of- speech tagging (Ling et al., 2015a), and sentiment analysis (Wang et al., 2016), leading to improved performance over training the models from scratch. The core idea underlying the training methodology of all the above embedding models is that “a word is characterized by the company it keeps” (Firth, 1957). For instance, the continuous bag-of-words (CBOW) model of word2vec predicts a randomly selected word in a sentence using the other words in the sentence as a context. The context words are typically treated equally by these models. However, clearly some words in the context will be more predictive of the masked word than others, and intuitively, should be weighted more heavily. A promising remedy that treats each context word differently is the attention mechanism (Vaswani et al., 2017), which enables learning the relative importance of input features for the prediction of the desired output. Attention has become the de facto standard and state-of-the-art in modern NLP for a range of different tasks (Zhang et al., 2019; Devlin et al., 2019; Dai et al., 2019). However, to date, there have been only a few attempts to employ it to improve the performance and interpretability of word embedding models (Ling et al., 2015b; Schick and Sch¨utze, 2019). 1.1 Contributions In this paper, we propose the Attention Word Embedding (AWE), a new word embedding model that integrates the attention mechanism into the CBOW model of word2vec. AWE consists of two components. First, AWE uses a variant of self-attention …

Content truncated to 500 words