Meta-Learning for Fast Cross-Lingual Adaptation in Dependency Parsing

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 8503 - 8520 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Meta-Learning for Fast Cross-Lingual Adaptation in Dependency Parsing Anna Langedijk⋆,1, Verna Dankers1,2, Phillip Lippe1, Sander Bos1, Bryan Cardenas Guevara1, Helen Yannakoudakis3, and Ekaterina Shutova1 1Institute for Logic, Language and Computation, University of Amsterdam 2Institute of Language, Cognition and Computation, University of Edinburgh 3Department of Informatics, King’s College London Abstract Meta-learning, or learning to learn, is a tech- nique that can help to overcome resource scarcity in cross-lingual NLP problems, by en- abling fast adaptation to new tasks. We ap- ply model-agnostic meta-learning (MAML) to the task of cross-lingual dependency parsing. We train our model on a diverse set of lan- guages to learn a parameter initialization that can adapt quickly to new languages. We find that meta-learning with pre-training can signif- icantly improve upon the performance of lan- guage transfer and standard supervised learn- ing baselines for a variety of unseen, typolog- ically diverse, and low-resource languages, in a few-shot learning setup. 1 Introduction The field of natural language processing (NLP) has seen substantial performance improvements due to large-scale language model pre-training (Devlin et al., 2019). Whilst providing an informed starting point for subsequent task-specific fine-tuning, such models still require large annotated training sets for the task at hand (Yogatama et al., 2019). This limits their applicability to a handful of languages for which such resources are available and leads to an imbalance in NLP technology’s quality and availability across linguistic communities. Aiming to address this problem, recent research has fo- cused on the development of multilingual sentence encoders, such as multilingual BERT (mBERT) (Devlin et al., 2019) and XLM-R (Conneau et al., 2020), trained on as many as 93 languages. Such pre-trained multilingual encoders enable zero-shot transfer of task-specific models across languages (Wu and Dredze, 2019), offering a possible solu- tion to resource scarcity. Zero-shot transfer, how- ⋆ Corresponding author: annalangedijk@gmail.com. ever, is most successful among typologically sim- ilar, high-resource languages, and less so for lan- guages distant from the training languages and in resource-lean scenarios (Lauscher et al., 2020). This stresses the need to develop techniques for fast cross-lingual model adaptation, that can transfer knowledge across a wide range of typologically diverse languages with limited supervision. In this paper, we focus on the task of universal dependency (UD) parsing and present a novel ap- proach for effective and resource-lean cross-lingual parser adaptation via meta-learning, requiring only a small number of training examples per language (which are easy to obtain even for low-resource languages). Meta-learning is a learning paradigm that leverages previous experience from a set of tasks to solve a new task efficiently. As our goal is fast cross-lingual model adaptation, we focus on optimization-based meta-learning, where the main objective is to find a set of initial parameters from which rapid adaption to a variety of different tasks becomes possible (Hospedales et al., 2020). Optimization-based meta-learning has been suc- cessfully …

Content truncated to 500 words