Meta-Learning for Effective Multi-task and Multilingual Modelling

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 3600–3612 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 3600 Meta-Learning for Effective Multi-task and Multilingual Modelling Ishan Tarunesh1 Sushil Khyalia1 Vishwajeet Kumar2 Ganesh Ramakrishnan1 Preethi Jyothi1 1 Indian Institute of Technology Bombay 2 IBM India Research Lab {ishan, sushil, ganesh, pjyothi}@cse.iitb.ac.in vishk024@in.ibm.com Abstract Natural language processing (NLP) tasks (e.g. question-answering in English) benefit from knowledge of other tasks (e.g., named entity recognition in English) and knowledge of other languages (e.g., question-answering in Spanish). Such shared representations are typ- ically learned in isolation, either across tasks or across languages. In this work, we pro- pose a meta-learning approach to learn the in- teractions between both tasks and languages. We also investigate the role of different sam- pling strategies used during meta-learning. We present experiments on five different tasks and six different languages from the XTREME multilingual benchmark dataset (Hu et al., 2020). Our meta-learned model clearly im- proves in performance compared to competi- tive baseline models that also include multi- task baselines. We also present zero-shot eval- uations on unseen target languages to demon- strate the utility of our proposed model. 1 Introduction Multi-task and multilingual learning are both prob- lems of long standing interest in natural language processing. Leveraging data from multiple tasks and/or additional languages to benefit a target task is of great appeal, especially when the target task has limited resources. When it comes to multiple tasks, it is well-known from prior work on multi- task learning (Liu et al., 2019b; Kendall et al., 2018; Liu et al., 2019a; Yang and Hospedales, 2017) that jointly learning a model across tasks can benefit the tasks mutually. For multiple languages, the ability of deep learning models to learn effective embeddings has led to their use for joint learning of models across languages (Conneau et al., 2020; Conneau and Lample, 2019; Artetxe and Schwenk, 2019); learning cross-lingual embeddings to aid languages in limited resource settings is of grow- ing interest (Kumar et al., 2019; Wang et al., 2017; Adams et al., 2017). Let us say we had access to M tasks across N different languages - c.f. Table 1 that outlines such a matrix of tasks and languages from the XTREME benchmark (Hu et al., 2020). How do we perform effective joint learning across tasks and languages? Are there specific tasks or languages that need to be sampled more frequently for effective joint training? Can such sampling strategies be learned from the data? In this work, we adopt a meta-learning approach for efficiently learning parameters in a shared pa- rameter space across multiple tasks and multiple languages. Our chosen tasks are question an- swering (QA), natural language inference (NLI), paraphrase identification (PA), part-of-speech tag- ging (POS) and named entity recognition (NER). The tasks were chosen to enable us to employ a gamut of different types of language representa- tions needed to tackle problems in NLP. In Fig- ure 1, we illustrate the …

Content truncated to 500 words