Meta-Embedding Sentence Representation for Textual Similarity

Proceedings of Recent Advances in Natural Language Processing, pages 465–473, Varna, Bulgaria, Sep 2–4, 2019. https://doi.org/10.26615/978-954-452-056-4_055 465 Meta-Embedding Sentence Representation for Textual Similarity Amir Hazem1 Nicolas Hernandez1 1 LS2N - UMR CNRS 6004, Universit´e de Nantes, France {Amir.Hazem,Nicolas.Hernandez}@univ-nantes.fr Abstract Word embedding models are now widely used in most NLP applications. Despite their effectiveness, there is no clear evi- dence about the choice of the most appro- priate model. It often depends on the na- ture of the task and on the quality and size of the used data sets. This remains true for bottom-up sentence embedding models. However, no straightforward investigation has been conducted so far. In this paper, we propose a systematic study of the im- pact of the main word embedding models on sentence representation. By contrast- ing in-domain and pre-trained embedding models, we show under which conditions they can be jointly used for bottom-up sen- tence embeddings. Finally, we propose the first bottom-up meta-embedding rep- resentation at the sentence level for tex- tual similarity. Significant improvements are observed in several tasks including question-to-question similarity, paraphras- ing and next utterance ranking. 1 Introduction According to Enkvist (1987): ”a model is a sim- plified representation of reality. It is simplified be- cause it aims at reproducing a selection of rele- vant elements of reality rather than all of real- ity at once.”. If several word embedding mod- els (Mikolov et al., 2013a; Pennington et al., 2014; Yin and Sch¨utze, 2016; Arora et al., 2017) capture a selection of relevant features, differ- ent embedding sets can cover different charac- teristics which can also be complementary (Yin and Sch¨utze, 2016). In order to capture a wide range of features, it is useful to perform models combination (ensemble models). The representa- tion of longer pieces of texts such as sentences, by an element-wise sum of their word embed- dings has recently shown promising results and outperformed sophisticated models in several tex- tual similarity tasks (Mikolov et al., 2013a; Arora et al., 2017). This representation, also known as bottom-up sentence embeddings, is greatly af- fected by the choice of word embedding mod- els. In this paper, we propose a systematic study of the impact of word embedding models on bottom-up sentence representation for textual sim- ilarity. We report the results of the main individual pre-trained embedding models that are publicly available as well as embedding models trained on in-domain data sets. Finally, we contrast multiple ensemble models and propose the first bottom-up meta-embedding sentence representa- tion for textual similarity. We evaluate the differ- ent approaches on four tasks that is: question-to- question similarity (SemEval 2016/2017), textual entailment (SemEval 2014), paraphrasing (Sick) and next utterance ranking (NUR) and show under which conditions meta-embeddings can be benefi- cial to bottom-up sentence-based approaches. 2 Related Work Embedding models at the word level representa- tions have been widely explored in many appli- cations (Bengio et al., 2003; Collobert and We- ston, 2008; Mikolov et al., 2013a; Pennington et al., 2014; Bojanowski et al., 2016). Naturally, they have …

Content truncated to 500 words