Language Embeddings for Typology and Cross-lingual Transfer Learning

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 7210–7225 August 1–6, 2021. ©2021 Association for Computational Linguistics 7210 Language Embeddings for Typology and Cross-lingual Transfer Learning Dian Yu∗, Taiqi He∗ and Kenji Sagae University of California, Davis {dianyu, tqhe, sagae}@ucdavis.edu Abstract Cross-lingual language tasks typically require a substantial amount of annotated data or par- allel translation data. We explore whether language representations that capture relation- ships among languages can be learned and subsequently leveraged in cross-lingual tasks without the use of parallel data. We gener- ate dense embeddings for 29 languages using a denoising autoencoder, and evaluate the em- beddings using the World Atlas of Language Structures (WALS) and two extrinsic tasks in a zero-shot setting: cross-lingual dependency parsing and cross-lingual natural language in- ference1. 1 Introduction Recent efforts to leverage multilingual datasets in language modeling (Conneau and Lample, 2019; Devlin et al., 2019) and machine translation (John- son et al., 2017; Lu et al., 2018) highlight the po- tential of multilingual models that can perform well across various languages, including ones for which training sets are scarce. Most of the current multilingual research focuses on learning invariant representations or removing language-specific fea- tures after training (Libovický et al., 2020; Bjerva and Augenstein, 2021). Despite recent advances, there are still limitations. Previous work has shown that similar languages can benefit from sharing pa- rameters, but less similar languages do not help (Zoph et al., 2016; Pires et al., 2019). However, in spite of some interests in typology (Ponti et al., 2019), identifying similar languages is nontrivial, especially for less studied ones. In addition, as Zhao et al. (2019) suggest, learning invariant rep- resentations can actually harm model performance. ∗Equal contribution. 1Our learned language embeddings and code avail- able at https://github.com/DianDYu/language_ embeddings Therefore, in order to leverage language agnostic and language specific information effectively, we propose to generate language representations and examine the interactions among different language representations. One way to represent language identity within a multilingual model is the use of language codes, or dense vectors representing language embed- dings. If languages are represented with vectors that capture cross-lingual similarities and differ- ences across different dimensions, this information can guide a multilingual model regarding what and how much of the information in the model should be shared among specific languages. Much of the previous research focused on generating language embeddings using prior knowledge such as word or- der (Ammar et al., 2016; Littell et al., 2017), using a parallel corpus (Bjerva et al., 2019b; Östling and Tiedemann, 2017), and using language codes as an indicator to distinguish input and output words in a shared vocabulary into different languages (John- son et al., 2017; Conneau and Lample, 2019). In contrast, our work focuses on generating and us- ing language embeddings more effectively as soft- sharing (de Lhoneux et al., 2018) of parameters among various languages in a single model. Fur- thermore, we are motivated by a …

Content truncated to 500 words