Polyglot Neural Language Models: A Case Study in Cross-Lingual Phonetic Representation Learning

Proceedings of NAACL-HLT 2016, pages 1357–1366, San Diego, California, June 12-17, 2016. c⃝2016 Association for Computational Linguistics Polyglot Neural Language Models: A Case Study in Cross-Lingual Phonetic Representation Learning Yulia Tsvetkov Sunayana Sitaram Manaal Faruqui Guillaume Lample Patrick Littell David Mortensen Alan W Black Lori Levin Chris Dyer Language Technologies Institute Carnegie Mellon University Pittsburgh, PA, 15213, USA {ytsvetko,ssitaram,mfaruqui,glample,plittell,dmortens,awb,lsl,cdyer}@cs.cmu.edu Abstract We introduce polyglot language models, re- current neural network models trained to pre- dict symbol sequences in many different lan- guages using shared representations of sym- bols and conditioning on typological infor- mation about the language to be predicted. We apply these to the problem of modeling phone sequences—a domain in which univer- sal symbol inventories and cross-linguistically shared feature representations are a natural fit. Intrinsic evaluation on held-out perplexity, qualitative analysis of the learned representa- tions, and extrinsic evaluation in two down- stream applications that make use of phonetic features show (i) that polyglot models bet- ter generalize to held-out data than compara- ble monolingual models and (ii) that polyglot phonetic feature representations are of higher quality than those learned monolingually. 1 Introduction Nearly all existing language model (LM) architec- tures are designed to model one language at a time. This is unsurprising considering the historical im- portance of count-based models in which every sur- face form of a word is a separately modeled entity (English cat and Spanish gato would not likely ben- efit from sharing counts). However, recent mod- els that use distributed representations—in partic- ular models that share representations across lan- guages (Hermann and Blunsom, 2014; Faruqui and Dyer, 2014; Huang et al., 2015; Lu et al., 2015, inter alia)—suggest universal models applicable to mul- tiple languages are a possibility. This paper takes a step in this direction. We introduce polyglot language models: neural network language models that are trained on and ap- plied to any number of languages. Our goals with these models are the following. First, to facilitate data and parameter sharing, providing more training resources to languages, which is especially valuable in low-resource settings. Second, models trained on diverse languages with diverse linguistic properties will better be able to learn naturalistic representa- tions that are less likely to “overfit” to a single lin- guistic outlier. Finally, polyglot models offer con- venience in a multilingual world: a single model re- places dozens of different models. Exploration of polyglot language models at the sentence level—the traditional domain of language modeling—requires dealing with a massive event space (i.e., the union of words across many lan- guages). To work in a more tractable domain, we evaluate our model on phone-based language mod- eling, the modeling sequences of sounds, rather than words. We choose this domain since a common assumption of many theories of phonology is that all spoken languages construct words from a finite inventory of phonetic symbols (represented conve- niently as the elements of the the International Pho- netic Alphabet; IPA) which are distinguished by language-universal features (e.g., place and manner of articulation, voicing status, etc.). …

Content truncated to 500 words