Learning Word Embeddings for Low-Resource Languages by PU Learning

Proceedings of NAACL-HLT 2018, pages 1024–1034 New Orleans, Louisiana, June 1 - 6, 2018. c⃝2018 Association for Computational Linguistics Learning Word Embeddings for Low-resource Languages by PU Learning Chao Jiang University of Virginia cj7an@virginia.edu Cho-Jui Hsieh University of California Davis chohsieh@ucdavis.edu Hsiang-Fu Yu Amazon rofuyu@cs.utexas.edu Kai-Wei Chang University of California Los Angeles kwchang@cs.ucla.edu Abstract Word embedding is a key component in many downstream applications in processing natu- ral languages. Existing approaches often as- sume the existence of a large collection of text for learning effective word embedding. However, such a corpus may not be avail- able for some low-resource languages. In this paper, we study how to effectively learn a word embedding model on a corpus with only a few million tokens. In such a situa- tion, the co-occurrence matrix is sparse as the co-occurrences of many word pairs are unob- served. In contrast to existing approaches of- ten only sample a few unobserved word pairs as negative samples, we argue that the zero entries in the co-occurrence matrix also pro- vide valuable information. We then design a Positive-Unlabeled Learning (PU-Learning) approach to factorize the co-occurrence matrix and validate the proposed approaches in four different languages. 1 Introduction Learning word representations has become a fundamental problem in processing natural lan- guages. These semantic representations, which map a word into a point in a linear space, have been widely applied in downstream applications, including named entity recognition (Guo et al., 2014), document ranking (Nalisnick et al., 2016), sentiment analysis (Irsoy and Cardie, 2014), ques- tion answering (Antol et al., 2015), and image cap- tioning (Karpathy and Fei-Fei, 2015). Over the past few years, various approaches have been proposed to learn word vectors (e.g., (Pennington et al., 2014; Mikolov et al., 2013a; Levy and Goldberg, 2014b; Ji et al., 2015)) based on co-occurrence information between words ob- served on the training corpus. The intuition behind this is to represent words with similar vectors if they have similar contexts. To learn a good word embedding, most approaches assume a large col- lection of text is freely available, such that the es- timation of word co-occurrences is accurate. For example, the Google Word2Vec model (Mikolov et al., 2013a) is trained on the Google News dataset, which contains around 100 billion to- kens, and the GloVe embedding (Pennington et al., 2014) is trained on a crawled corpus that contains 840 billion tokens in total. However, such an as- sumption may not hold for low-resource languages such as Inuit or Sindhi, which are not spoken by many people or have not been put into a digital format. For those languages, usually, only a lim- ited size corpus is available. Training word vectors under such a setting is a challenging problem. One key restriction of the existing approaches is that they often mainly rely on the word pairs that are observed to co-occur on the training data. When the size of the text corpus is small, most word pairs are unobserved, resulting in an ex- tremely sparse …

Content truncated to 500 words