Multilingual LLMs are Better Cross-lingual In-context Learners with Alignment

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 6292–6307 July 9-14, 2023 ©2023 Association for Computational Linguistics Multilingual LLMs are Better Cross-lingual In-context Learners with Alignment Eshaan Tanwar DTU, India eshaantanwar2000@gmail.com Subhabrata Dutta IIT Delhi, India subha0009@gmail.com Manish Borthakur IIT Delhi, India mt6190493@iitd.ac.in Tanmoy Chakraborty IIT Delhi, India tanchak@iitd.ac.in Abstract In-context learning (ICL) unfolds as large lan- guage models become capable of inferring test labels conditioned on a few labeled samples without any gradient update. ICL-enabled large language models provide a promising step forward toward bypassing recurrent an- notation costs in a low-resource setting. Yet, only a handful of past studies have explored ICL in a cross-lingual setting, in which the need for transferring label-knowledge from a high-resource language to a low-resource one is immensely crucial. To bridge the gap, we provide the first in-depth analysis of ICL for cross-lingual text classification. We find that the prevalent mode of selecting random input- label pairs to construct the prompt-context is severely limited in the case of cross-lingual ICL, primarily due to the lack of alignment in the input as well as the output spaces. To mitigate this, we propose a novel prompt con- struction strategy – Cross-lingual In-context Source-Target Alignment (X-InSTA). With an injected coherence in the semantics of the input examples and a task-based alignment across the source and target languages, X-InSTA is able to outperform random prompt selection by a large margin across three different tasks using 44 different cross-lingual pairs. 1 Introduction The emergence of large-scale, pretrained, Transformer-based language models (LLMs) has marked the commencement of an avant-garde era in NLP. Departing from the traditional methods of neural language learning with temporally separated training-testing phases for downstream tasks, pretrained LLMs have shown the ability to infer labels from test inputs conditioned on the training data within a single pass. This is known as In-context learning – an LLM is prompted ET and SD contributed equally. ET and SD designed the experiments. ET and MB ran the experiments. SD and TC wrote the paper. TC mentored the project. with a few input-output pairs from the training data (commonly referred to as demonstrations) followed by the test input; for generative tasks (summarization, text-to-code, chain-of-thought reasoning, etc.) the LLM is then required to produce an output; for classification tasks, the probabilities of the next tokens predicted by the LLM are mapped to the label space. All of this is done without updating the parameters of the LLM. In-context learning is particularly promising for two different aspects. Firstly, it reduces the need for task-specific training data, and thus, the cost of human annotation. Secondly, while the LLM was trained in a compute-intensive environment, the removal of the need for task-specific gradient- based weight updates can significantly reduce the carbon footprint of automated NLP/NLU since the inference-time compute-necessity is orders of magnitude smaller than that of the training/finetuning phases. Multiple recent advancements have been proposed to optimize the ICL ability of the LLMs (Lin …

Content truncated to 500 words