Minimax and Neyman–Pearson Meta-Learning for Outlier Languages

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 1245–1260 August 1–6, 2021. ©2021 Association for Computational Linguistics 1245 Minimax and Neyman–Pearson Meta-Learning for Outlier Languages Edoardo Maria Ponti1,2∗ Rahul Aralikatte3∗ Disha Shrivastava1,4 Siva Reddy1,2 Anders Søgaard3 1Mila – Quebec Artificial Intelligence Institute 2McGill University 3University of Copenhagen 4University of Montreal 1{edoardo-maria.ponti,siva.reddy,disha.shrivastava}@mila.quebec 3{rahul,soegaard}@di.ku.dk Abstract Model-agnostic meta-learning (MAML) has been recently put forth as a strategy to learn resource-poor languages in a sample-efficient fashion. Nevertheless, the properties of these languages are often not well represented by those available during training. Hence, we argue that the i.i.d. assumption ingrained in MAML makes it ill-suited for cross-lingual NLP. In fact, under a decision-theoretic frame- work, MAML can be interpreted as minimis- ing the expected risk across training languages (with a uniform prior), which is known as Bayes criterion. To increase its robustness to outlier languages, we create two variants of MAML based on alternative criteria: Minimax MAML reduces the maximum risk across lan- guages, while Neyman–Pearson MAML con- strains the risk in each language to a maxi- mum threshold. Both criteria constitute fully differentiable two-player games. In light of this, we propose a new adaptive optimiser solv- ing for a local approximation to their Nash equilibrium. We evaluate both model vari- ants on two popular NLP tasks, part-of-speech tagging and question answering. We report gains for their average and minimum perfor- mance across low-resource languages in zero- and few-shot settings, compared to joint multi- source transfer and vanilla MAML. The code for our experiments is available at https:// github.com/rahular/robust-maml. 1 Introduction Knowledge transfer is ubiquitous in machine learn- ing because of the general scarcity of annotated data (Pratt, 1993; Caruana, 1997; Ruder, 2019, in- ter alia). A prominent example thereof is transfer from resource-rich languages to resource-poor lan- guages (Wu and Dredze, 2019; Ponti et al., 2019b; Ruder et al., 2019). Recently, Model-Agnostic ∗Equal contribution Meta-Learning (MAML; Finn et al., 2017) has come to the fore as a promising paradigm: it ex- plicitly trains neural models that adapt to new lan- guages quickly by extrapolating from just a few an- notated data points (Gu et al., 2018; Nooralahzadeh et al., 2020; Wu et al., 2020; Li et al., 2020). MAML usually rests on the simplifying assump- tion that the source ‘tasks’ and the target ‘tasks’ are independent and identically distributed (hence- forth, i.i.d.). However, in practice most scenarios of cross-lingual transfer violate this assumption: train- ing languages documented in mainstream datasets do not reflect the cross-lingual variation, as they belong to a clique of few families, geographical ar- eas, and typological features (Bender, 2009; Joshi et al., 2020). Therefore, the majority of the world’s languages lies outside of such a clique. As training and evaluation languages differ in their joint dis- tribution, they are not exchangeable (Ponti, 2021; Orbanz, 2012, ch. 6). Therefore, there is no for- mal guarantee that MAML generalises to the very languages whose need for transfer is most critical. In this work, we interpret meta-learning within a decision-theoretic …

Content truncated to 500 words