Cross-Lingual Word Embeddings for Low-Resource Language Modeling

Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 937–947, Valencia, Spain, April 3-7, 2017. c⃝2017 Association for Computational Linguistics Cross-Lingual Word Embeddings for Low-Resource Language Modeling Oliver Adams,♠♥ Adam Makarucha,♠ Graham Neubig,♣ Steven Bird,♥♦ Trevor Cohn♥ ♠IBM Research – Australia ♥Computing and Information Systems, University of Melbourne ♣School of Computer Science, Carnegie Mellon University ♦International Computer Science Institute, University of California Berkeley oliver.adams@gmail.com, adamjm@au1.ibm.com, gneubig@cs.cmu.edu, {t.cohn,steven.bird}@unimelb.edu.au Abstract Most languages have no established writ- ing system and minimal written records. However, textual data is essential for nat- ural language processing, and particularly important for training language models to support speech recognition. Even in cases where text data is missing, there are some languages for which bilingual lexicons are available, since creating lexicons is a fun- damental task of documentary linguistics. We investigate the use of such lexicons to improve language models when tex- tual training data is limited to as few as a thousand sentences. The method involves learning cross-lingual word embeddings as a preliminary step in training monolin- gual language models. Results across a number of languages show that language models are improved by this pre-training. Application to Yongning Na, a threatened language, highlights challenges in deploy- ing the approach in real low-resource en- vironments. 1 Introduction Most of the world’s languages are not actively written, even languages with an official writing system (Bird, 2011). This limits the available textual data to small quantities of phonemic tran- scriptions prepared by linguists. Since phone- mic transcription is time-consuming, such data is scarce. This makes language modeling, which is a key tool for facilitating speech recognition of these languages, a difficult challenge. One of the touted advantages of neural network lan- guage models (NNLMs) is their ability to model sparse data (Bengio et al., 2003; Gandhe et al., 2014). However, despite the success of NNLMs on large datasets (Mikolov et al., 2010; Sutskever et al., 2011; Graves, 2013), it remains unclear whether their advantages transfer to scenarios with extremely limited amounts of data. Appropriate initialization of parameters in neu- ral network frameworks has been shown to be ben- eficial across a wide variety of domains, includ- ing speech recognition, where unsupervised pre- training of deep belief networks was instrumental in attaining breakthrough performance (Hinton et al., 2012). Neural network approaches to a range of NLP problems have also been aided by ini- tialization with word embeddings trained on large amounts of unannotated text (Frome et al., 2013; Zhang et al., 2014; Lau and Baldwin, 2016). How- ever, in the case of extremely low-resource lan- guages we do not have the luxury of this unanno- tated text. As a remedy to this problem we focus on cross- lingual word embeddings (CLWEs), which learn word embeddings using information from multi- ple languages. Recent advances in CLWEs have shown that high quality embeddings can be learnt even in the absence of bilingual corpora by har- nessing bilingual lexicons (Gouws and Søgaard, 2015; Duong …

Content truncated to 500 words