PRO-CS : An Instance-Based Prompt Composition Technique for Code-Switched Tasks

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10243 - 10255 December 7-11, 2022 ©2022 Association for Computational Linguistics PRO-CS : An Instance-Based Prompt Composition Technique for Code-Switched Tasks Srijan Bansal∗ Suraj Tripathi∗ Sumit Agarwal∗ Teruko Mitamura Eric Nyberg {srijanb, surajt, sumita, teruko, en09} @andrew.cmu.edu Language Technologies Institute, Carnegie Mellon University Abstract Code-switched (CS) data is ubiquitous in to- day’s globalized world, but the dearth of anno- tated datasets in code-switching poses a signifi- cant challenge for learning diverse tasks across different language pairs. Parameter-efficient prompt-tuning approaches conditioned on frozen language models have shown promise for transfer learning in limited-resource setups. In this paper, we propose a novel instance- based prompt composition technique, PRO- CS, for CS tasks that combine language and task knowledge. We compare our approach with prompt-tuning and fine-tuning for code- switched tasks on 10 datasets across 4 language pairs. Our model outperforms the prompt- tuning approach by significant margins across all datasets and outperforms or remains at par with fine-tuning by using just 0.18% of total parameters. We also achieve competitive re- sults when compared with the fine-tuned model in the low-resource cross-lingual and cross- task setting, indicating the effectiveness of our approach to incorporate new code-switched tasks. Our code and models will be available at https://github.com/srijan-bansal/PRO-CS 1 Introduction Code-Switching (CS) is the phenomenon of shift- ing from one language to another in the same con- text, usually used in an informal way of speaking or writing (Sitaram et al., 2019; Jose et al., 2020). Within the scope of this paper, we have focused on the most commonly used form of code-switching which happens at the intra-sentential level. With the advent of social media platforms and the global rise in multilingual speakers, people generally con- verse in more than one language and often switch from one language to another (Parshad et al., 2016). Code-switching can go beyond the mere insertion of borrowed words, fillers, and phrases, and in- clude morphological and grammatical mixing. It is ∗Equal contribution Figure 1: Different settings of PRO-CS 1, (a) prompt- composition for En-Hi POS code-switched task with source languages (En, Hi) and task (POS) prompts (b) prompt-composition for cross-lingual transfer to En-Bn POS from related source language Hi-POS prompt, (c) prompt-composition for cross-task transfer from En and Hi SA to En-Hi HS. frequently used to convey stronger emotions or to express one’s ideas precisely (Sitaram et al., 2019). Code-switching has gained attention in the NLP community due to a plethora of unstructured text data available that require an understanding of in- termixing of two languages in the same sentence. Understanding the meaning of such text relies on 1Text in (a, b) He/She is very cute, (c) I hate Hindus. The languages shown are English (En), Hindi (Hi) and Bengali (Bn). The tasks used are Part of Speech (POS), Sentiment Analysis (SA), and Hate Speech (HS). 10243 subtasks such as part-of-speech (POS) tagging, sen- timent analysis (SA), and hate speech (HS), and the characteristic of CS …

Content truncated to 500 words