Evaluating Unsupervised Dutch Word Embeddings as a Linguistic Resource

Evaluating Unsupervised Dutch Word Embeddings as a Linguistic Resource St´ephan Tulkens, Chris Emmery, Walter Daelemans CLiPS, University of Antwerp firstname.lastname@uantwerpen.be Abstract Word embeddings have recently seen a strong increase in interest as a result of strong performance gains on a variety of tasks. However, most of this research also underlined the importance of benchmark datasets, and the difficulty of constructing these for a variety of language-specific tasks. Still, many of the datasets used in these tasks could prove to be fruitful linguistic resources, allowing for unique observations into language use and variability. In this paper we demonstrate the performance of multiple types of embeddings, created with both count and prediction-based architectures on a variety of corpora, in two language-specific tasks: relation evaluation, and dialect identification. For the latter, we compare unsupervised methods with a traditional, hand-crafted dictionary. With this research, we provide the embeddings themselves, the relation evaluation task benchmark for use in further research, and demonstrate how the benchmarked embeddings prove a useful unsupervised linguistic resource, effectively used in a downstream task. Keywords: word embeddings, benchmarking, word2vec, SPPMI, language variation, dialect identification 1. Introduction The strong variability of language use within, and across textual media (Collins et al., 1977; Linell, 1982) has on many occasions been marked as an important challenge for research in the area of computational linguistics (Resnik, 1999; Rosenfeld, 2000), in particular in applications to social media (Gouws et al., 2011). Formal and infor- mal varieties, as well as an abundance of deviations from grammar and spelling conventions in the latter, drastically complicate computationally interpreting the meaning of, and relations between words. This task of understand- ing lies at the heart of natural language processing (NLP). Neural-network-based language models such as the models in word2vec have recently gained strong interest in NLP due to the fact that they improved state-of-the-art perfor- mance on a variety of tasks in the field. Given these devel- opments, we found it surprising that only one set of word embeddings has been publicly released for Dutch (Al-Rfou et al., 2013), which does not offer sufficiently large dimen- sionality for state-of-the-art performance. The primary goal of this research is thus evaluating word embeddings derived from several popular Dutch corpora and the impact of these sources on their quality, specifically focusing on problems characteristic for Dutch. Word embeddings—being an un- supervised technique—cannot be easily evaluated without comparing performance in some downstream task. There- fore, we present two novel benchmarking tasks of our own making: a relation identification task analogous to previ- ous evaluations on English, in which the quality of differ- ent kinds of word embeddings is measured, and a dialect identification task which measures the usefulness of word embeddings as a linguistic resource for Dutch in particular. In the literature, there has been some debate on the effec- tiveness of prediction-based embeddings when compared to more classical count-based embedding models (Baroni et al., 2014). As such, we train both count- (SPPMI) and prediction-based (SGNS) models, and compare them to previous efforts in …

Content truncated to 500 words