Word Embedding Evaluation in Downstream Tasks and Semantic Analogies

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 4828–4834 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 4828 Word Embedding Evaluation in Downstream Tasks and Semantic Analogies Joaquim Santos, Bernardo Consoli, Renata Vieira Pontifical Catholic University of Rio Grande do Sul (PUCRS) Porto Alegre, Brazil {joaquim.santos, bernardo.consoli}@acad.pucrs.br, renata.vieira@pucrs.br Abstract Language Models have long been a prolific area of study in the field of Natural Language Processing (NLP). One of the newer kinds of language models, and some of the most used, are Word Embeddings (WE). WE are vector space representations of a vocabulary learned by a non-supervised neural network based on the context in which words appear. WE have been widely used in downstream tasks in many areas of study in NLP. These areas usually use these vector models as a feature in the processing of textual data. This paper presents the evaluation of newly released WE models for the Portuguese language, trained with a corpus composed of 4.9 billion tokens. The first evaluation presented an intrinsic task in which WEs had to correctly build semantic and syntactic relations. The second evaluation presented an extrinsic task in which the WE models were used in two downstream tasks: Named Entity Recognition and Semantic Similarity between Sentences. Our results show that a diverse and comprehensive corpus can often outperform a larger, less textually diverse corpus, and that passing the text in parts to the WE generating algorithm may cause loss of quality. Keywords: Language Models, Word Embeddings, Intrinsic Evaluation, Extrinsic Evaluation 1. Introduction Language Models (LM) are an oft studied area of Natural Language Processing (NLP) in current literature. LMs were first developed based on the study of linguistic rules that could not be generalized to languages other than the one they were created for. In the eighties, another generation of LMs were developed, based on statistical models that used conditional probabilities to analyze n-gram sequences. This new strategy became known as N-grams Models, or Statis- tical LMs (Jing and Xu, 2019). Yet another advancement occurred based on the work of Xu and Rudnicky (2000), which suggests the usage of Neural Networks (NN), more specifically Feedforward NNs, for LMs. The work of Ben- gio et al. (2003), based on this new advancement, presented a solution to the area’s perennial problem: the curse of di- mensionality. From there, the usage of Recurrent Neural Networks started becoming more prevalent, which resulted in the adoption of vector space models for word representa- tion, called Word Embeddings (WE) (Mikolov et al., 2009; Mikolov et al., 2010; Mikolov et al., 2013b). WEs can thus be described as a vector space representing words learned through the non-supervised training of NN. These vector spaces can also be called embeddings. Each embedding is the result of successive mathematical opera- tions that happen in the Hidden Layers of the NN, and add semantic meaning to the embeddings. WEs have been successfully applied to NLP solutions, but the degree of success is …

Content truncated to 500 words