Tweaks and Tricks for Word Embedding Disruptions

Proceedings of Recent Advances in Natural Language Processing, pages 460–464, Varna, Bulgaria, Sep 2–4, 2019. https://doi.org/10.26615/978-954-452-056-4_054 460 Tweaks and Tricks for Word Embedding Disruptions Amir Hazem1 Nicolas Hernandez1 1 LS2N - UMR CNRS 6004, Universit´e de Nantes, France {Amir.Hazem,Nicolas.Hernandez}@univ-nantes.fr Abstract Word embeddings are established as very effective models used in several NLP ap- plications. If they differ in their architec- ture and training process, they often ex- hibit similar properties and remain vector space models with continuously-valued dimensions describing the observed data. The complexity resides in the developed strategies for learning the values within each dimensional space. In this paper, we introduce the concept of disruption which we define as a side effect of the training process of embedding models. Disrup- tions are viewed as a set of embedding val- ues that are more likely to be noise than effective descriptive features. We show that dealing with disruption phenomenon is of a great benefit to bottom-up sentence embedding representation. By contrast- ing several in-domain and pre-trained em- bedding models, we propose two simple but very effective tweaking techniques that yield strong empirical improvements on textual similarity task. 1 Introduction Word embedding models are now a standard in many NLP applications. If the choice of the most appropriate model is not always straightforward, context word representation is at the core of each model and the performance is closely related to how well the context is exploited. In this pa- per, we introduce the notion of disruption, a phe- nomenon caused by the training process of word embedding models. Disruptions are a set of ex- treme embedding values that are more likely to be noise than reliable features. We consider this phenomenon as a negative side effect closely re- lated to the data and to the training optimization decisions. If we observe the Gaussian distribu- tion of word embedding dimensional-values, we notice a set of high positive and negative values of which the percentage varies from one embed- ding model to another. In the context of vector- space models where each word is represented as a point in the N-dimensional Euclidean space, dis- ruptions may drastically affect word’s position and so create unbalanced embedding values. If nor- malization tends to smooth this effect, our experi- ments reveal that detecting disruptions and adjust- ing them is far more efficient than a standard nor- malization. We show that dealing with disruptions is of a substantial benefit to bottom-up sentence embedding representation. A bottom-up sentence representation is a weighted sum of the embedding vectors of its constituent words. This simple approach turned out to be very competitive in many NLP appli- cations (Wieting et al., 2016; Arora et al., 2017) and outperformed several advanced RNNs and LSTM-based models of which the performance heavily depends on the quality and the large size of the training data set (Socher et al., 2011; Le and Mikolov, 2014; Kiros et al., 2015; Pagliar- dini et al., 2018). By contrast to sophisticated approaches, bottom-up sentence embedding mod- els are less …

Content truncated to 500 words