Nearest Neighbour Few-Shot Learning for Cross-lingual Classification

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1745–1753 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 1745 Nearest Neighbour Few-Shot Learning for Cross-lingual Classification M Saiful Bari ∗ §¶, Batool Haider §, Saab Mansour§ § Amazon AI ¶ Nanyang Technological University, Singapore ¶bari0001@e.ntu.edu.sg §{bhaider, saabm}@amazon.com Abstract Even though large pre-trained multilingual models (e.g. mBERT, XLM-R) have led to sig- nificant performance gains on a wide range of cross-lingual NLP tasks, success on many downstream tasks still relies on the availabil- ity of sufficient annotated data. Traditional fine-tuning of pre-trained models using only a few target samples can cause over-fitting. This can be quite limiting as most languages in the world are under-resourced. In this work, we investigate cross-lingual adaptation using a simple nearest neighbor few-shot (< 15 samples) inference technique for classification tasks. We experiment using a total of 16 dis- tinct languages across two NLP tasks- XNLI and PAWS-X. Our approach consistently im- proves traditional fine-tuning using only a handful of labeled samples in target locales. We also demonstrate its generalization capabil- ity across tasks. 1 Introduction The rise of massively pre-trained multilingual lan- guage models (LM)1 (Lample and Conneau, 2019; Conneau et al., 2020; Chi et al., 2020; Luo et al., 2020; Xue et al., 2020) has significantly improved cross-lingual generalization across many languages (Wu and Dredze, 2019; Pires et al., 2019; K et al., 2020; Keung et al., 2019). Recent work on zero- shot cross-lingual adaptation (Bari et al., 2020; Fang et al., 2020; Pfeiffer et al., 2020), in the ab- sence of labelled target data, has also demonstrated impressive performance gains. Despite these suc- cesses, however, there still remains a sizeable gap between supervised and zero-shot performances. On the other hand, when limited target language data are available (i.e few-shot setting), traditional fine-tuning of large pre-trained models can cause over-fitting (Perez and Wang, 2017). ∗Work done while Saiful was interning at Amazon AI 1We loosely use the term LM to describe unsupervised pretrained models including Masked-LMs and Causal-LMs One way to deal with the scarcity of annotated data is to augment synthetic data using techniques like paraphrasing (Gao et al., 2020; Du et al., 2020), word translation (Xie et al., 2018; Mohiuddin and Joty, 2020; Mohiuddin et al., 2020), machine trans- lation (Sennrich et al., 2015), data-augmentation (Ding et al., 2020; Liu et al., 2021; Laskar et al., 2020; Ding et al., 2020) and/or data-diversification (Nguyen et al., 2019; Mohiuddin et al., 2021; Bari et al., 2021). Few-shot learning, on the other hand, deals with handling out-of-distribution (OOD) gen- eralization problems using only a small amount of data (Koch, 2015; Vinyals et al., 2016; Jake Snell, 2017; Santoro et al., 2017; Chelsea Finn, 2017). In this setup, the model is evaluated over few-shot tasks, such that the model learns to generalize to new data (query set) using only a hand full of la- beled samples (support set). In a cross-lingual few-shot setup, the model learns cross-lingual features to generalize to …

Content truncated to 500 words