Hyper-X: A Unified Hypernetwork for Multi-Task Multilingual Transfer

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 7934 - 7949 December 7-11, 2022 ©2022 Association for Computational Linguistics Hyper-X: A Unified Hypernetwork for Multi-Task Multilingual Transfer Ahmet Üstün1, Arianna Bisazza1, Gosse Bouma1, Gertjan van Noord1, Sebastian Ruder2 1University of Groningen 2Google Research a.ustun@rug.nl Abstract Massively multilingual models are promising for transfer learning across tasks and languages. However, existing methods are unable to fully leverage training data when it is available in dif- ferent task-language combinations. To exploit such heterogeneous supervision, we propose Hyper-X, a single hypernetwork that unifies multi-task and multilingual learning with effi- cient adaptation. This model generates weights for adapter modules conditioned on both tasks and language embeddings. By learning to combine task and language-specific knowledge, our model enables zero-shot transfer for un- seen languages and task-language combina- tions. Our experiments on a diverse set of lan- guages demonstrate that Hyper-X achieves the best or competitive gain when a mixture of mul- tiple resources is available, while being on par with strong baselines in the standard scenario. Hyper-X is also considerably more efficient in terms of parameters and resources compared to methods that train separate adapters. Finally, Hyper-X consistently produces strong results in few-shot scenarios for new languages, showing the versatility of our approach beyond zero-shot transfer.1 1 Introduction Transfer learning across languages and tasks has long been an important focus in NLP (Ruder et al., 2019). Recent advances in massively multilingual transformers (MMTs; Devlin et al., 2019; Conneau et al., 2020) show great success in this area. A benefit of such models is their ability to transfer task-specific information in a high-resource source language to a low-resource target language (Fig- ure 1, 1 ). Alternatively, such models can lever- age knowledge from multiple tasks for potentially stronger generalization (Figure 1, 2 ). Over time, many research communities have been developing resources for specific languages 1Our code for Hyper-X will be released at https://github.com/ahmetustun/hyperx NER en Pre-trained Model Pre-trained Model Fine-tuned Model ar tr Fine-tuned Model ar tr POS en Single-Task Pre-trained Model Fine-tuned Model ar tr ar tr NER POS en en Multi-Task Pre-trained Model Fine-tuned Model tr NER POS ar tr NER ar POS en en Mixed-Language Multi-Task Figure 1: Experimental settings of different (zero-shot) cross-lingual transfer scenarios. Single-task (1) is the standard setting; multi-task (2) enables cross-task trans- fer. Mixed-language multi-task (3) additionally allows leveraging task data from multiple source languages for different tasks. of focus (Strassel and Tracey, 2016; Nivre et al., 2018; Wilie et al., 2020). In practice, it is thus common for data to be available for different tasks in a mixture of different languages. For instance, in addition to English data for both POS tagging and Named Entity Recognition (NER), a treebank with POS annotation may be available for Turkish, while NER data may be available for Arabic. This example is illustrated in Figure 1, 3 . In contrast to existing cross-lingual transfer paradigms such as single-task zero-shot transfer (Hu et al., 2020) or few-shot …

Content truncated to 500 words