Polyglot Prompt: Multilingual Multitask Prompt Training

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9919 - 9935 December 7-11, 2022 ©2022 Association for Computational Linguistics Polyglot Prompting: Multilingual Multitask Prompt Training Jinlan Fu NUS jinlan@nus.edu.sg See-Kiong Ng NUS seekiong@nus.edu.sg Pengfei Liu CMU & Inspired Cognition stefanpengfei@gmail.com Abstract This paper aims for a potential architectural im- provement for multilingual learning and asks: Can different tasks from different languages be modeled in a monolithic framework, i.e. with- out any task/language-specific module? The benefit of achieving this could open new doors for future multilingual research, including al- lowing systems trained on low resources to be further assisted by other languages as well as other tasks. We approach this goal by devel- oping a learning framework named Polyglot Prompting to exploit prompting methods for learning a unified semantic space for different languages and tasks with multilingual prompt engineering. We performed a comprehensive evaluation of 6 tasks, namely topic classifi- cation, sentiment classification, named entity recognition, question answering, natural lan- guage inference, and summarization, covering 24 datasets and 49 languages. The experimen- tal results demonstrated the efficacy of multilin- gual multitask prompt-based learning and led to inspiring observations. We also present an interpretable multilingual evaluation method- ology and show how the proposed framework, multilingual multitask prompt training, works. We release all datasets prompted in the best setting and code. 1 1 Introduction The emergence of multilingual pre-trained lan- guage models (Xue et al., 2021; Liu et al., 2020; Conneau et al., 2020; Conneau and Lample, 2019) enables different languages to be represented in a unified semantic space. As a result, a fine-tuned model of a data-rich language such as English can achieve decent transfer (e.g., zero-shot) per- formance in geographically, syntactically, or pho- netically similar languages (Malaviya et al., 2017). The insufficient features learned by languages un- der lower-resource settings can thus be compen- 1https://github.com/jinlanfu/Polyglot_Prompt QA-en QA-zh NLI-ja NLI-fr Sum-es Sum-de PLM QA Sum NLI QA (a) Generic PLM PLM (b) MT (c) MLMT PLM PLM Sum NLI Figure 1: Application of prompt technology in three dif- ferent scenarios: Generic, Multitask (MT), Multilingual Multitask (MLMT). QA, Sum, and NLI represent differ- ent tasks, namely question answering, summarization, and natural language inference here. PLM represents pre-trained language model, and “zh”, “en”, “fr”, “ja”, “de”,“es” denote different languages. sated through the higher-resource languages shared with them. Despite the preliminary success in the low- resource scenarios using shared knowledge across languages in multilingual language models (Wang et al., 2019b; Liu et al., 2019; K et al., 2020), the cross-lingual transfers have mostly occurred only within the boundary of the same task or similar tasks. Can more conceptually-diverse tasks from different languages communicate together? While researchers have made some preliminary progress towards this direction (Yang et al., 2016; Lin et al., 2018; Hasan et al., 2021; Dong et al., 2015; Ma- habadi et al., 2021), the scope of the “different” tasks had remained relatively narrow, such as limit- ing to tasks with the same sequence labeling form (Yang et al., …

Content truncated to 500 words