Resources to Examine the Quality of Word Embedding Models Trained on n-Gram Data

Proceedings of the 22nd Conference on Computational Natural Language Learning (CoNLL 2018), pages 423–432 Brussels, Belgium, October 31 - November 1, 2018. c⃝2018 Association for Computational Linguistics 423 Resources to Examine the Quality of Word Embedding Models Trained on n-Gram Data ´Abel Elekes Adrian Englhardt Martin Sch¨aler Klemens B¨ohm Karlsruhe Institute of Technology (KIT), Karlsruhe, Germany {abel.elekes, adrian.englhardt, martin.schaeler, klemens.boehm}@kit.edu Abstract Word embeddings are powerful tools that fa- cilitate better analysis of natural language. However, their quality highly depends on the resource used for training. There are various approaches relying on n-gram corpora, such as the Google n-gram corpus. However, n- gram corpora only offer a small window into the full text – 5 words for the Google corpus at best. This gives way to the concern whether the extracted word semantics are of high qual- ity. In this paper, we address this concern with two contributions. First, we provide a resource containing 120 word-embedding models – one of the largest collection of embedding mod- els. Furthermore, the resource contains the n- gramed versions of all used corpora, as well as our scripts used for corpus generation, model generation and evaluation. Second, we de- fine a set of meaningful experiments allow- ing to evaluate the aforementioned quality dif- ferences. We conduct these experiments us- ing our resource to show its usage and signifi- cance. The evaluation results confirm that one generally can expect high quality for n-grams with n ≥ 3. 1 Introduction Motivation. Word embedding approaches like Word2Vec (Mikolov et al., 2013b) or Glove (Pen- nington et al., 2014) are powerful tools for the semantic analysis of natural language. One can train them on arbitrary text corpora. Each word in the corpus is mapped to a d-dimensional vec- tor. These vectors feature the semantic similarity and analogy properties, as follows. Semantic sim- ilarity means that representations of words used in a similar context tend to be close to each other in the vector space. The analogy property can be de- scribed by the example that ”man” is to ”woman” like ”king” to ”queen” (Mikolov et al., 2013b,a; Jansen, 2017). These properties have been applied in numer- ous approaches (Mitra and Craswell, 2017) like sentiment analysis (Tang et al., 2014), irony detec- tion (Reyes et al., 2012), out-of-vocabulary word classification (Ma and Zhang, 2015), or semantic shift detection (Hamilton et al., 2016a; Martinez- Ortiz et al., 2016). One prerequisite when creating high-quality embedding models is a good train- ing corpus. To this end, many approaches use the Google n-gram corpus (Hellrich and Hahn, 2016; Pyysalo et al., 2013; Kim et al., 2014; Martinez- Ortiz et al., 2016; Kulkarni et al., 2016, 2015; Hamilton et al., 2016b). It also is the largest cur- rently available corpus with historic data and ex- ists for several languages. It incorporates over 5 million books from the last centuries split into n- grams (Michel et al., 2015). n-grams are text seg- ments separated into pieces consisting of n words each. The fragmentation of a corpus …

Content truncated to 500 words