Learning Language Representations for Typology Prediction

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2529–2535 Copenhagen, Denmark, September 7–11, 2017. c⃝2017 Association for Computational Linguistics Learning Language Representations for Typology Prediction Chaitanya Malaviya and Graham Neubig and Patrick Littell Language Technologies Institute Carnegie Mellon University {cmalaviy,gneubig,pwl}@cs.cmu.edu Abstract One central mystery of neural NLP is what neural models “know” about their subject matter. When a neural machine transla- tion system learns to translate from one language to another, does it learn the syn- tax or semantics of the languages? Can this knowledge be extracted from the sys- tem to fill holes in human scientific knowl- edge? Existing typological databases con- tain relatively full feature specifications for only a few hundred languages. Ex- ploiting the existence of parallel texts in more than a thousand languages, we build a massive many-to-one neural machine translation (NMT) system from 1017 lan- guages into English, and use this to pre- dict information missing from typological databases. Experiments show that the pro- posed method is able to infer not only syn- tactic, but also phonological and phonetic inventory features, and improves over a baseline that has access to information about the languages’ geographic and phy- logenetic neighbors.1 1 Introduction Linguistic typology is the classification of human languages according to syntactic, phonological, and other classes of features, and the investiga- tion of the relationships and correlations between these classes/features. This study has been a sci- entific pursuit in its own right since the 19th cen- tury (Greenberg, 1963; Comrie, 1989; Nichols, 1992), but recently typology has borne practical fruit within various subfields of NLP, particularly on problems involving lower-resource languages. 1Code and learned vectors are available at http:// github.com/chaitanyamalaviya/lang-reps Figure 1: Learning representations from mul- tilingual neural MT for typology classification. (Model MTBOTH) Typological information from sources like the World Atlas of Language Structures (WALS) (Dryer and Haspelmath, 2013), has proven use- ful in many NLP tasks (O’Horan et al., 2016), such as multilingual dependency parsing (Ammar et al., 2016), generative parsing in low-resource settings (Naseem et al., 2012; T¨ackstr¨om et al., 2013), phonological language modeling and loan- word prediction (Tsvetkov et al., 2016), POS- tagging (Zhang et al., 2012), and machine trans- lation (Daiber et al., 2016). However, the needs of NLP tasks differ in many ways from the needs of scientific typology, and ty- pological databases are often only sparsely pop- ulated, by necessity or by design.2 In NLP, on the other hand, what is important is having a rela- tively full set of features for the particular group of languages you are working on. This mis- match of needs has motivated various proposals to reconstruct missing entries, in WALS and other databases, from known entries (Daum´e III and Campbell, 2007; Daum´e III, 2009; Coke et al., 2016; Littell et al., 2017). In this study, we examine whether we can 2For example, each chapter of WALS aims to provide a statistically balanced set of languages over language families and geographical areas, and so many languages are left …

Content truncated to 500 words