Word Embedding Evaluation and Combination

Word Embeddings Evaluation and Combination Sahar Ghannay1, Benoit Favre2, Yannick Est`eve1 and Nathalie Camelin1 1LIUM - University of Le Mans, 72000, Le Mans, France 2Aix-Marseille Universit´e, CNRS, LIF UMR 7279, 13000, Marseille, France Abstract Word embeddings have been successfully used in several natural language processing tasks (NLP) and speech processing. Different approaches have been introduced to calculate word embeddings through neural networks. In the literature, many studies focused on word embedding evaluation, but for our knowledge, there are still some gaps. This paper presents a study focusing on a rigorous comparison of the performances of different kinds of word embeddings. These performances are evaluated on different NLP and linguistic tasks, while all the word embeddings are estimated on the same training data using the same vocabulary, the same number of dimensions, and other similar characteristics. The evaluation results reported in this paper match those in the literature, since they point out that the improvements achieved by a word embedding in one task are not consistently observed across all tasks. For that reason, this paper investigates and evaluates approaches to combine word embeddings in order to take advantage of their complementarity, and to look for the effective word embeddings that can achieve good performances on all tasks. As a conclusion, this paper provides new perceptions of intrinsic qualities of the famous word embedding families, which can be different from the ones provided by works previously published in the scientific literature. Keywords: Word embeddings, benchmarking, speech processing, natural language processing 1. Introduction Word embeddings are projections in a continuous space of words supposed to preserve the semantic and syntac- tic similarities between them. They have been shown to be a great asset for several Natural Language Processing (NLP) tasks, like part-of-speech tagging, chunking, named entity recognition, semantic role labeling, syntactic pars- ing (Bansal et al., 2014a; Turian et al., 2010; Collobert et al., 2011), and also for speech processing: for instance, word embeddings were recently involved in spoken lan- guage understanding (Mesnil et al., 2015), in detection of errors in automatic transcriptions, and in calibration of con- fidence measures provided by an automatic speech recog- nition system (Ghannay et al., 2015). These word representations were introduced through the construction of neural language models (Bengio et al., 2003; Schwenk, 2013). Different approaches have been proposed to compute them from large corpora. They in- clude neural networks (Collobert et al., 2011; Mikolov et al., 2013a; Pennington et al., 2014), dimensionality reduc- tion on the word co-occurrence matrix (Lebret and Col- lobert, 2013), and explicit representation in terms of the context in which words appear (Levy and Goldberg, 2014). One particular hypothesis behind word embeddings is that they are generic representations that shall suit most appli- cations. Many studies have focused on the evaluation of word em- beddings intrinsic quality, as well as their impact when they are used as input of systems. Turian et al. (Turian et al., 2010) evaluate different types of word representations and their concatenation on the chunking and named entity …

Content truncated to 500 words