From Phonology to Syntax: Unsupervised Linguistic Typology at Different Levels with Language Embeddings

Proceedings of NAACL-HLT 2018, pages 907–916 New Orleans, Louisiana, June 1 - 6, 2018. c⃝2018 Association for Computational Linguistics From Phonology to Syntax: Unsupervised Linguistic Typology at Different Levels with Language Embeddings Johannes Bjerva Department of Computer Science University of Copenhagen Denmark bjerva@di.ku.dk Isabelle Augenstein Department of Computer Science University of Copenhagen Denmark augenstein@di.ku.dk Abstract A core part of linguistic typology is the clas- sification of languages according to linguistic properties, such as those detailed in the World Atlas of Language Structure (WALS). Doing this manually is prohibitively time-consuming, which is in part evidenced by the fact that only 100 out of over 7,000 languages spoken in the world are fully covered in WALS. We learn distributed language representations, which can be used to predict typological prop- erties on a massively multilingual scale. Ad- ditionally, quantitative and qualitative analy- ses of these language embeddings can tell us how language similarities are encoded in NLP models for tasks at different typological levels. The representations are learned in an unsuper- vised manner alongside tasks at three typolog- ical levels: phonology (grapheme-to-phoneme prediction, and phoneme reconstruction), mor- phology (morphological inflection), and syn- tax (part-of-speech tagging). We consider more than 800 languages and find significant differences in the language rep- resentations encoded, depending on the tar- get task. For instance, although Norwegian Bokm˚al and Danish are typologically close to one another, they are phonologically distant, which is reflected in their language embed- dings growing relatively distant in a phonolog- ical task. We are also able to predict typolog- ical features in WALS with high accuracies, even for unseen language families. 1 Introduction For more than two and a half centuries, linguistic typologists have studied languages with respect to their structural and functional properties (Haspel- math, 2001; Velupillai, 2012). Although typol- ogy has a long history (Herder, 1772; Gabelentz, 1891; Greenberg, 1960, 1974; Dahl, 1985; Com- rie, 1989; Haspelmath, 2001; Croft, 2002), com- putational approaches have only recently gained popularity (Dunn et al., 2011; W¨alchli, 2014; ¨Ostling, 2015; Cotterell and Eisner, 2017; Asgari and Sch¨utze, 2017; Malaviya et al., 2017; Bjerva and Augenstein, 2018). One part of traditional ty- pological research can be seen as assigning sparse explicit feature vectors to languages, for instance manually encoded in databases such as the World Atlas of Language Structures (WALS, Dryer and Haspelmath, 2013). A recent development which can be seen as analogous to this is the process of learning distributed language representations in the form of dense real-valued vectors, often re- ferred to as language embeddings (Tsvetkov et al., 2016; ¨Ostling and Tiedemann, 2017; Malaviya et al., 2017). We hypothesise that these language embeddings encode typological properties of lan- guage, reminiscent of the features in WALS, or even of parameters in Chomsky’s Principles and Parameters framework (Chomsky, 1993). In this paper, we model languages in deep neu- ral networks using language embeddings, consid- ering three typological levels: phonology, mor- phology and syntax. We consider four NLP tasks to be representative of these levels: grapheme-to- phoneme prediction and …

Content truncated to 500 words