BAD-X: Bilingual Adapters Improve Zero-Shot Cross-Lingual Transfer

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1791 - 1799 July 10-15, 2022 ©2022 Association for Computational Linguistics BAD-X: Bilingual Adapters Improve Zero-Shot Cross-Lingual Transfer Marinela Parovi´c1 Goran Glavaš2 Ivan Vuli´c1 Anna Korhonen1 1Language Technology Lab, TAL, University of Cambridge 2CAIDAS, University of Würzburg {mp939,iv250,alk23}@cam.ac.uk goran.glavas@uni-wuerzburg.de Abstract Adapter modules enable modular and efficient zero-shot cross-lingual transfer, where current state-of-the-art adapter-based approaches learn specialized language adapters (LAs) for indi- vidual languages. In this work, we show that it is more effective to learn bilingual language pair adapters (BAs) when the goal is to op- timize performance for a particular source- target transfer direction. Our novel BAD-X adapter framework trades off some modularity of dedicated LAs for improved transfer per- formance: we demonstrate consistent gains in three standard downstream tasks, and for the majority of evaluated low-resource languages. 1 Introduction Massively multilingual Transformers (MMTs) such as mBERT (Devlin et al., 2019), XLM-R (Con- neau et al., 2020), and mT5 (Xue et al., 2021) have dominated research in multilingual NLP and cross-lingual transfer recently. Pretrained on large amounts of unlabelled data in 100+ languages, they have been shown to achieve impressive perfor- mance for a wide range of languages and tasks, and in zero-shot cross-lingual transfer in particular (Wu and Dredze, 2019; K et al., 2019). However, their representational capacity is known to be limited by the curse of multilinguality: a trade-off between the language coverage and model capacity (Conneau et al., 2020), which typically favors high-resource languages. Their limitations are thus especially pronounced in low-resource scenarios, in transfer between distant languages and towards resource- poor target languages (Hu et al., 2020; Lauscher et al., 2020; Ansell et al., 2021, inter alia). A standard approach to zero-shot cross-lingual transfer with MMTs (i) fine-tunes the full MMT on task-specific data in the source language and then (ii) applies it directly to make predictions in the target language (Hu et al., 2020). On top of the expensive fine-tuning of the entire large model, this standard procedure also does not ‘prepare’ the MMT to excel at a particular target language or for a particular source-target transfer direction. This has been alleviated through modular parameter-efficient adaptations of the MMTs (Bapna and Firat, 2019; Philip et al., 2020; He et al., 2021) which bypass full fine-tuning, most prominently through lightweight adapters (Rebuffi et al., 2017; Houlsby et al., 2019): additional train- able parameters inserted into the MMT’s layers. They have recently been used for language and task specialization of the MMTs (Pfeiffer et al., 2020b), offering improved and more efficient zero- shot cross-lingual transfer. Previous work (Pfeiffer et al., 2020b; Üstün et al., 2020, 2021; Vidoni et al., 2020; Ansell et al., 2021, inter alia) focused on creating: 1) dedicated lan- guage adapters (LAs) for each individual language, and 2) individual task adapters (TAs). Creating single-language LAs enables a very modular ap- proach to cross-lingual transfer, where a source language LA (used in training) can …

Content truncated to 500 words