Zero-shot Cross-lingual Transfer of Prompt-based Tuning with a Unified Multilingual Prompt

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11488 - 11497 December 7-11, 2022 ©2022 Association for Computational Linguistics Zero-shot Cross-lingual Transfer of Prompt-based Tuning with a Unified Multilingual Prompt Lianzhe Huang†∗, Shuming Ma‡, Dongdong Zhang‡, Furu Wei‡ and Houfeng Wang† †MOE Key Lab of Computational Linguistics, Peking University ‡Microsoft Research Asia {hlz, wanghf}@pku.edu.cn {shumma, dozhang, fuwei}@microsoft.com Abstract Prompt-based tuning has been proven effec- tive for pretrained language models (PLMs). While most of the existing work focuses on the monolingual prompts, we study the mul- tilingual prompts for multilingual PLMs, es- pecially in the zero-shot cross-lingual setting. To alleviate the effort of designing different prompts for multiple languages, we propose a novel model that uses a unified prompt for all languages, called UniPrompt. Different from the discrete prompts and soft prompts, the unified prompt is model-based and language- agnostic. Specifically, the unified prompt is initialized by a multilingual PLM to pro- duce language-independent representation, af- ter which is fused with the text input. During inference, the prompts can be pre-computed so that no extra computation cost is needed. To collocate with the unified prompt, we pro- pose a new initialization method for the target label word to further improve the model’s trans- ferability across languages. Extensive exper- iments show that our proposed methods can significantly outperform the strong baselines across different languages. We release data and code to facilitate future research1. 1 Introduction Pre-trained language models (PLMs) have been proven to be successful in various downstream tasks (Devlin et al., 2019; Yang et al., 2019; Con- neau et al., 2020). Prompt-tuning is one of the effective ways to induce knowledge from PLMs to improve downstream task performance especially when the labeled data is not sufficient (Brown et al., 2020; Gao et al., 2021; Le Scao and Rush, 2021; Zhao and Schütze, 2021). The essence of prompt- tuning is to precisely design task input structure so that it can imitate the pre-training procedure of ∗Contribution during internship at Microsoft. 1The data and the code of this paper are available at https://github.com/mojave-pku/UniPrompt. It’s [mask]. Food is great. (It’s [mask]. This restaurant is terrible.) 😊 (positive) [mask] ❌ average ✔ good ❌ bad { Model Train 它是 [mask]。这家餐馆很糟糕。 Test 🙁 (negative) [mask] ❌ ⼀般的 (average) ❌ 好的 (good) ✔ 差的 (bad) { Figure 1: An example of zero-shot cross- lingual transfer of prompt-based tuning. The underline part of the input with [mask] token is template. PLMs and better induce the knowledge from them. For example, to classify the sentiment polarity of the source sentence “Food is great”, a template “It’s [mask].” is constructed before the source input where the accurate label is masked. In this way, the sentiment-related words like ‘good’, ‘bad’, and ‘average’ is predicted at the masked position with probabilities on the target side, over which a ver- balizer is leveraged to project the final sentiment labels. Previously, most work on prompt-based tun- ing (Gao et al., 2021; Zhang et al., 2021) mainly considers the monolingual …

Content truncated to 500 words