Finely Tuned, 2 Billion Token Based Word Embeddings for Portuguese

Finely Tuned, 2 Billion Token Based Word Embeddings for Portuguese Jo˜ao Rodrigues and Ant´onio Branco University of Lisbon NLX-Natural Language and Speech Group, Department of Informatics Faculdade de Ciˆencias Campo Grande, 1749-016 Lisboa, Portugal {joao.rodrigues, antonio.branco}@di.fc.ul.pt Abstract A distributional semantics model — also known as word embeddings — is a major asset for any language as the research results reported in the literature have consistently shown that it is instrumental to improve the performance of a wide range of applications and processing tasks for that language. In this paper, we describe the development of an advanced distributional model for Portuguese, with the largest vocabulary and the best evaluation scores published so far. This model was made possible by resorting to new languages resources we recently developed: to a much larger training corpus than before and to a more sophisticated evaluation supported by new and more fine-grained evaluation tasks and data sets. We also indicate how the new language resource reported on here is being distributed and where it can be obtained for free under a most permissive license. Keywords: distributional semantics, word embeddings, word analogy, lexical similarity, conceptual categorization, Portuguese. 1. Introduction Distributional semantics assumes that the frequency of con- texts in which expressions occur helps to capture impor- tant syntactic and semantic properties of these expressions (Garvin, 1962). Exploring distributional models has led to advances in a range of natural language processing tasks, from dialog systems (Chen et al., 2014) to question answer- ing (Bordes et al., 2014), among many others. A distributional model is a major language resource for any language as it is instrumental to enhance the performance of many applications and processing tasks for that language. Recent work concerned with distributional models for Por- tuguese contributed with the creation and public release of a free distributional model for this language that resorts to a considerably large corpus, for the training, and to a stan- dard analogy test set, for the evaluation and tuning of this model (Rodrigues et al., 2016). Since then that corpus has been greatly expanded by our group and is now a very large data set, reaching more than 2 billion tokens. More recently, finer tuning of distributional models for Por- tuguese became possible with the creation and distribution of a wide range of data sets by our group that support the evaluation of lexical similarity and conceptual categoriza- tion tasks, and that are comparable in size and domain to mainstream datasets with the same purpose for other lan- guages (Querido et al., 2017). The aim of the present paper is to report on the develop- ment, and its free distribution, of a more advanced distri- butional model for Portuguese, with the largest vocabulary and the best evaluation scores published so far, made pos- sible by a much larger training corpus than before and by a more sophisticated evaluation supported by new and more fine-grained evaluation tasks and data sets. In Section 2., we discuss previous work in the literature related to the …

Content truncated to 500 words