A Closer Look at Few-Shot Crosslingual Transfer: The Choice of Shots Matters

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 5751–5767 August 1–6, 2021. ©2021 Association for Computational Linguistics 5751 A Closer Look at Few-Shot Crosslingual Transfer: The Choice of Shots Matters Mengjie Zhao1* Yi Zhu2* Ehsan Shareghi3, 2 Ivan Vuli´c2 Roi Reichart4 Anna Korhonen2 Hinrich Sch¨utze1 1CIS, LMU Munich 2LTL, University of Cambridge 3Department of Data Science & AI, Monash University 4Faculty of Industrial Engineering and Management, Technion, IIT mzhao@cis.lmu.de, {yz568,iv250,alk23}@cam.ac.uk, ehsan.shareghi@monash.edu, roiri@technion.ac.il Abstract Few-shot crosslingual transfer has been shown to outperform its zero-shot counterpart with pretrained encoders like multilingual BERT. Despite its growing popularity, little to no at- tention has been paid to standardizing and an- alyzing the design of few-shot experiments. In this work, we highlight a fundamental risk posed by this shortcoming, illustrating that the model exhibits a high degree of sensitivity to the selection of few shots. We conduct a large- scale experimental study on 40 sets of sampled few shots for six diverse NLP tasks across up to 40 languages. We provide an analysis of success and failure cases of few-shot transfer, which highlights the role of lexical features. Additionally, we show that a straightforward full model finetuning approach is quite effec- tive for few-shot transfer, outperforming sev- eral state-of-the-art few-shot approaches. As a step towards standardizing few-shot crosslin- gual experimental designs, we make our sam- pled few shots publicly available.1 1 Introduction Multilingual pretrained encoders like multilingual BERT (mBERT; Devlin et al. (2019)) and XLM- R (Conneau et al., 2020) are the top performers in crosslingual tasks such as natural language in- ference (Conneau et al., 2018), document clas- sification (Schwenk and Li, 2018; Artetxe and Schwenk, 2019), and argument mining (Toledo- Ronen et al., 2020). They enable transfer learn- ing through language-agnostic representations in crosslingual setups (Hu et al., 2020). A widely explored transfer scenario is zero-shot crosslingual transfer (Pires et al., 2019; Conneau and Lample, 2019; Artetxe and Schwenk, 2019), * Equal contribution. 1Code and resources are available at https://github. com/fsxlt where a pretrained encoder is finetuned on abun- dant task data in the source language (e.g., English) and then directly evaluated on target-language test data, achieving surprisingly good performance (Wu and Dredze, 2019; Hu et al., 2020). However, there is evidence that zero-shot performance reported in the literature has large variance and is often not re- producible (Keung et al., 2020a; Rios et al., 2020); the results in languages distant from English fall far short of those similar to English (Hu et al., 2020; Liang et al., 2020). Lauscher et al. (2020) stress the importance of few-shot crosslingual transfer instead, where the encoder is first finetuned on a source language and then further finetuned with a small amount (10–100) of examples (few shots) of the target lan- guage. The few shots substantially improve model performance of the target language with negligi- ble annotation costs (Garrette and Baldridge, 2013; Hedderich et al., 2020). In this work, however, we …

Content truncated to 500 words