Multilingual Grammar Induction with Continuous Language Identification

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 5728–5733, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 5728 Multilingual Grammar Induction with Continuous Language Identification∗ Wenjuan Han†, Ge Wang†, Yong Jiang‡, Kewei Tu† †School of Information Science and Technology, ShanghaiTech University, Shanghai, China ‡Alibaba Group {hanwj,wangge,tukw}@shanghaitech.edu.cn yongjiang.jy@alibaba-inc.com Abstract The key to multilingual grammar induction is to couple grammar parameters of differ- ent languages together by exploiting the sim- ilarity between languages. Previous work re- lies on linguistic phylogenetic knowledge to specify similarity between languages. In this work, we propose a novel universal grammar induction approach that represents language identities with continuous vectors and employs a neural network to predict grammar param- eters based on the representation. Without any prior linguistic phylogenetic knowledge, we automatically capture similarity between languages with the vector representations and softly tie the grammar parameters of different languages. In our experiments, we apply our approach to 15 languages across 8 language families and subfamilies in the Universal De- pendency Treebank dataset, and we observe substantial performance gain on average over monolingual and multilingual baselines. 1 Introduction Human languages bear striking resemblance at the syntactic level in spite of their diversity on the surface, as many studies have revealed (Green- berg, 1963; Hawkins, 2014). This fact provides the basis for multilingual grammar induction which tries to simultaneously induce grammars of mul- tiple languages. Intuitively, one can couple gram- mar parameters of different languages with simi- lar typology and learn them simultaneously. How- ever, the lacking of measures of language similar- ity prevents this idea from being further exploited in practice. Previous work in multilingual grammar induc- tion either does not consider language similar- ity measures (Iwata et al., 2010) or models lan- ∗The first and second authors contributed equally. The third author contributed to this work when at ShanghaiTech University. The fourth author is the corresponding author. guage similarity based on linguistic phylogeny (Berg-Kirkpatrick and Klein, 2010). The phylo- genetic knowledge, however, could be mislead- ing in measuring language similarity. For exam- ple, English and German are both Germanic lan- guages, but English exhibits dominant Subject- Verb-Object (SVO) word order while German does not. In this paper, we propose a novel approach to multilingual grammar induction. Our induction model represents language identities as continu- ous vectors (i.e., language embeddings) and em- ploys a neural network to predict the grammar pa- rameters of each language based on its embed- ding. The neural network parameters are univer- sally shared across languages, which softly tie the grammar parameters of different languages. The language embeddings and the neural network pa- rameters are trained with a standard grammar in- duction objective without any guidance from prior linguistic phylogenetic knowledge. We also intro- duce an auxiliary language identification task in which we predict the language identities of input sentences using the language embeddings. We evaluate our approach on corpora of 15 lan- guages across 8 language …

Content truncated to 500 words