X-METRA-ADA: Cross-lingual Meta-Transfer learning Adaptation to Natural Language Understanding and Question Answering

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3617–3632 June 6–11, 2021. ©2021 Association for Computational Linguistics 3617 X-METRA-ADA: Cross-lingual Meta-Transfer Learning Adaptation to Natural Language Understanding and Question Answering Meryem M'hamdi1∗, Doo Soon Kim2, Franck Dernoncourt2, Trung Bui2, Xiang Ren1 and Jonathan May1 1Information Sciences Institute, University of Southern California {meryem, xiangren, jonmay}@isi.edu 2Adobe Research {dkim, franck.dernoncourt, bui}@adobe.com Abstract Multilingual models, such as M-BERT and XLM-R, have gained increasing popularity, due to their zero-shot cross-lingual transfer learning capabilities. However, their gener- alization ability is still inconsistent for ty- pologically diverse languages and across dif- ferent benchmarks. Recently, meta-learning has garnered attention as a promising tech- nique for enhancing transfer learning under low-resource scenarios: particularly for cross- lingual transfer in Natural Language Under- standing (NLU). In this work, we propose X-METRA- ADA, a cross-lingual MEta-TRAnsfer learn- ing ADAptation approach for NLU. Our ap- proach adapts MAML, an optimization-based meta-learning approach, to learn to adapt to new languages. We extensively evaluate our framework on two challenging cross-lingual NLU tasks: multilingual task-oriented dialog and typologically diverse question answering. We show that our approach outperforms naive fine-tuning, reaching competitive performance on both tasks for most languages. Our analy- sis reveals that X-METRA-ADA can leverage limited data for faster adaptation. 1 Introduction Cross-lingual transfer learning is a technique used to adapt a model trained on a downstream task in a source language to directly generalize to the task in new languages. It aims to come up with common cross-lingual representations and leverages them to bridge the divide between resources to make any NLP application scale to multiple languages. This is particularly useful for data-scarce scenarios, as it reduces the need for API calls implied by machine translation or costly task-specific annotation for new languages. ∗Work was started while the first author was a research intern at Adobe. Figure 1: An overview of the X-METRA-ADA frame- work: we use English as the source and Spanish as the target language. The meta-train stage transfers from the source to the target languages, while the meta-adaptation further adapts the model to the target language. The application is few-shot if the test lan- guage is seen in any stage of X-METRA-ADA; or zero- shot if the test language is unseen. Transformer-based contextualized embeddings and their multilingual counterparts such as M- BERT (Devlin et al., 2019) have become popular as off-the-shelf representations for cross-lingual trans- fer learning. While these multilingual representa- tions exhibit some cross-lingual capability even for languages with low lexical overlap with English, the transfer quality is reduced for languages that exhibit different typological characteristics (Pires et al., 2019). The generalization of such representations has been extensively evaluated on traditional tasks such as Part-of-Speech (POS) tagging, Named Entity Recognition (NER) and Cross-lingual Document Classification (CLDC) (Ahmad et al., 2019; Wu and Dredze, 2019; Bari et al., 2020a; Schwenk and Li, 2018), with ever-growing open commu- nity annotation efforts like Universal Dependen- cies (Nivre et al., 2020) …

Content truncated to 500 words