Discrete and Soft Prompting for Multilingual Models

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 8547–8555 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 8547 Discrete and Soft Prompting for Multilingual Models Mengjie Zhao and Hinrich Schütze CIS, LMU Munich, Germany mzhao@cis.lmu.de Abstract It has been shown for English that discrete and soft prompting perform strongly in few- shot learning with pretrained language mod- els (PLMs). In this paper, we show that dis- crete and soft prompting perform better than finetuning in multilingual cases: Crosslingual transfer and in-language training of multilin- gual natural language inference. For exam- ple, with 48 English training examples, fine- tuning obtains 33.74% accuracy in crosslin- gual transfer, barely surpassing the majority baseline (33.33%). In contrast, discrete and soft prompting outperform finetuning, achiev- ing 36.43% and 38.79%. We also demonstrate good performance of prompting with training data in multiple languages other than English. 1 Introduction Prompting strongly outperforms finetuning (Devlin et al., 2019) when adapting pretrained language models (PLMs; Devlin et al. (2019); Conneau et al. (2020)) to downstream tasks in the low-resource regime (Brown et al., 2020; Schick and Schütze, 2021; Gao et al., 2020; Tam et al., 2021; Le Scao and Rush, 2021), i.e., few-shot learning, a more realistic scenario than having tens of thousands of annotations, even for English (Yu et al., 2018; Yin et al., 2020; Ram et al., 2021). In contrast to finetuning, which learns discrim- inative classifiers for tasks like natural language inference (NLI; Dagan et al. (2006); Bowman et al. (2015)), prompting reformulates the classification task to generative text-to-text (Raffel et al., 2020) or cloze-style (McCann et al., 2018; Brown et al., 2020) queries which are given to a PLM to answer. For example, the NLI task of assigning premise “They whinnied, eyes wide” and hypothesis “Their eyes were open wide” to class “entailment” can be reformulated as: They whinnied, eyes wide . Question: Their eyes were open wide ? Answer: . The PLM is requested to fill in, for the blank ( ), the word “yes”, which is mapped to “entailment”. Prompting makes a human description of the task available in learning. Also, “filling in the blank” is well aligned with the pretraining objec- tive (masked/autoregressive language modelling (Devlin et al., 2019; Radford et al., 2019; Yang et al., 2019)), likely to deliver better performance in few-shot learning (Ram et al., 2021). In this paper, we investigate the effectiveness of prompting in multilingual tasks, which – despite the success of prompting in English – is largely unexplored. We address two main research ques- tions: (RQ1) Does the strong few-shot performance of prompting transfer to other languages from En- glish? (RQ2) As the cost of few-shot non-English annotations is affordable (Garrette and Baldridge, 2013; Lauscher et al., 2020; Zhao et al., 2021), can we directly prompt PLMs in languages other than English or do we have to go through the (generally best resourced) intermediary of English? In this work, we systematically compare two popular prompting methods – discrete and soft prompting …

Content truncated to 500 words