Dirichlet-Smoothed Word Embeddings for Low-Resource Settings

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 3560–3565 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 3560 Dirichlet-Smoothed Word Embeddings for Low-Resource Settings Jakob Jungmaier, Nora Kassner, Benjamin Roth Center for Information and Language Processing, Ludwig-Maximilians-Universit¨at M¨unchen jakob.jungmaier@campus.lmu.de, {kassner, beroth}@cis.uni-muenchen.de Abstract Nowadays, classical count-based word embeddings using positive pointwise mutual information (PPMI) weighted co-occurrence matrices have been widely superseded by machine-learning-based methods like word2vec and GloVe. But these methods are usually applied using very large amounts of text data. In many cases, however, there is not much text data available, for example for specific domains or low-resource languages. This paper revisits PPMI by adding Dirichlet smoothing to correct its bias towards rare words. We evaluate on standard word similarity data sets and compare to word2vec and the recent state of the art for low-resource settings: Positive and Unlabeled (PU) Learning for word embeddings. The proposed method outperforms PU-Learning for low-resource settings and obtains competitive results for Maltese and Luxembourgish. Keywords: Statistical and Machine Learning Methods, Language Modelling, Neural Language Representation Models, Seman- tics, Less-Resourced/Endangered Languages 1. Introduction Word embeddings that capture the meaning of words as vectors are the basis of much of the recent progress in nat- ural language processing. Nowadays, the classical count- based way to obtain such word embeddings from a cor- pus by using positive pointwise mutual information (PPMI) weighted co-occurrence matrices has been widely super- seded by machine-learning-based methods like word2vec (Mikolov et al., 2013a; Mikolov et al., 2013b) and GloVe (Pennington et al., 2014). These methods are usually ap- plied using very large amounts of text data. But in many settings there is not much text data available, for example for specific domains or low-resource languages. Recent investigations by Konovalov and Tumunbayarova (2018) and Jiang et al. (2018) give rise to the conjecture that variants of the classical method to compute word em- beddings might be more suitable for smaller corpora than the machine-learning-based methods. Additionally, Levy et al. (2015) suggest that certain choices of system design and hyperparameters for computing word embeddings can even be more impactful than the choice of the underlying model architecture itself. In the spirit of these results, we propose to use a variant of the classical method for low-resource languages which tries to address a well-known problem of the PMI mea- sure: its bias towards rare words (Turney and Pantel, 2010; Levy et al., 2015; Jurafsky and Martin, 2018). Following an idea mentioned by Turney and Pantel (2010) and Juraf- sky and Martin (2018), we apply Dirichlet smoothing to weaken PMI’s bias towards rare words and compare it to the word2vec skip-gram with negative sampling (SGNS) and Positive and Unlabeled (PU) Learning for word em- beddings by Jiang et al. (2018). Models are trained on the English enwik9 corpus and eval- uated using standard word similarity data sets. We investi- gate the models’ performance for varying corpus size. In a case study, to demonstrate their practicability for …

Content truncated to 500 words