UDapter: Language Adaptation for Truly Universal Dependency Parsing

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 2302–2315, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 2302 UDapter: Language Adaptation for Truly Universal Dependency Parsing Ahmet ¨Ust¨un Arianna Bisazza Gosse Bouma Gertjan van Noord University of Groningen {a.ustun, a.bisazza, g.bouma, g.j.m.van.noord}@rug.nl Abstract Recent advances in multilingual dependency parsing have brought the idea of a truly uni- versal parser closer to reality. However, cross- language interference and restrained model ca- pacity remain major obstacles. To address this, we propose a novel multilingual task adap- tation approach based on contextual parame- ter generation and adapter modules. This ap- proach enables to learn adapters via language embeddings while sharing model parameters across languages. It also allows for an easy but effective integration of existing linguis- tic typology features into the parsing network. The resulting parser, UDapter, outperforms strong monolingual and multilingual baselines on the majority of both high-resource and low- resource (zero-shot) languages, showing the success of the proposed adaptation approach. Our in-depth analyses show that soft parame- ter sharing via typological features is key to this success.1 1 Introduction Monolingual training of a dependency parser has been successful when relatively large treebanks are available (Kiperwasser and Goldberg, 2016; Dozat and Manning, 2017). However, for many languages, treebanks are either too small or unavail- able. Therefore, multilingual models leveraging Universal Dependency annotations (Nivre et al., 2018) have drawn serious attention (Zhang and Barzilay, 2015; Ammar et al., 2016; de Lhoneux et al., 2018; Kondratyuk and Straka, 2019). Mul- tilingual approaches learn generalizations across languages and share information between them, making it possible to parse a target language with- out supervision in that language. Moreover, multi- lingual models can be faster to train and easier to maintain than a large set of monolingual models. 1Our code for UDapter is publicly available at https://github.com/ahmetustun/udapter However, scaling a multilingual model over a high number of languages can lead to sub-optimal results, especially if the training languages are typo- logically diverse. Often, multilingual neural mod- els have been found to outperform their monolin- gual counterparts on low- and zero-resource lan- guages due to positive transfer effects, but un- derperform for high-resource languages (Johnson et al., 2017; Arivazhagan et al., 2019; Conneau et al., 2020), a problem also known as “the curse of multilinguality”. Generally speaking, a multi- lingual model without language-specific supervi- sion is likely to suffer from over-generalization and perform poorly on high-resource languages due to limited capacity compared to the monolingual base- lines, as verified by our experiments on parsing. In this paper, we strike a good balance between maximum sharing and language-specific capacity in multilingual dependency parsing. Inspired by recently introduced parameter sharing techniques (Platanios et al., 2018; Houlsby et al., 2019), we propose a new multilingual parser, UDapter, that learns to modify its language-specific parameters including the adapter modules, as a function of language embeddings. This allows the model to share parameters across languages, ensuring gen- eralization and transfer ability, but also enables language-specific parameterization in a …

Content truncated to 500 words