Enhancing Cross-lingual Natural Language Inference by Prompt-learning from Cross-lingual Templates

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 1910 - 1923 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Enhancing Cross-lingual Natural Language Inference by Prompt-learning from Cross-lingual Templates Kunxun Qi1,3, Hai Wan1,3∗, Jianfeng Du2,4∗, Haolan Chen5 1 School of Computer Science and Engineering, Sun Yat-sen University, Guangzhou, China 2 Guangzhou Key Laboratory of Multilingual Intelligent Processing, Guangdong University of Foreign Studies, Guangzhou, China 3 Key Laboratory of Machine Intelligence and Advanced Computing (Sun Yat-sen University), Ministry of Education, China 4 Pazhou Lab, Guangzhou, China 5 Platform and Content Group, Tencent, Shenzhen, China Abstract Cross-lingual natural language inference (XNLI) is a fundamental task in cross-lingual natural language understanding. Recently this task is commonly addressed by pre-trained cross-lingual language models. Existing meth- ods usually enhance pre-trained language mod- els with additional data, such as annotated par- allel corpora. These additional data, however, are rare in practice, especially for low-resource languages. Inspired by recent promising re- sults achieved by prompt-learning, this paper proposes a novel prompt-learning based frame- work for enhancing XNLI. It reformulates the XNLI problem to a masked language modeling problem by constructing cloze-style questions through cross-lingual templates. To enforce correspondence between different languages, the framework augments a new question for every question using a sampled template in an- other language and then introduces a consis- tency loss to make the answer probability distri- bution obtained from the new question as sim- ilar as possible with the corresponding distri- bution obtained from the original question. Ex- perimental results on two benchmark datasets demonstrate that XNLI models enhanced by our proposed framework significantly outper- form original ones under both the full-shot and few-shot cross-lingual transfer settings. 1 Introduction Cross-lingual language understanding (XLU) plays a vital role in multilingual systems. It aims at train- ing a model in a source language which is then applied to other languages. Cross-lingual natural language inference (XNLI) is a challenge task for evaluating XLU (Conneau et al., 2018). Natural language inference (NLI) aims to determine the in- ferential relationship between the text of a premise and the text of a hypothesis while XNLI upgrades NLI to the cross-lingual scenarios. ∗ Corresponding authors: wanhai@mail.sysu.edu.cn, jfdu@gdufs.edu.cn <s>آﻣل أن أﺳﻣﻊ ﻣﻧك ﻗرﯾﺑﺎ</s> </s> Question:اﺗﻣﻧﻰ ان ﻧﺗﺣدث ﻗرﯾﺑﺎ? Answer: <MASK> </s> Premise Hypothesis Premise <s>آﻣل أن أﺳﻣﻊ ﻣﻧك ﻗرﯾﺑﺎ</s> </s><MASK>ﺳؤال: اﺗﻣﻧﻰ ان ﻧﺗﺣدث ﻗرﯾﺑﺎ؟ اﻟﺟواب:</s> Hypothesis Applying Arabic template to Arabic example: Applying English template to Arabic example: Training example in English with English template: <s> I hope to hear from you soon </s> </s> Question: I hope we talk soon ? Answer: <MASK> </s> Premise Hypothesis Figure 1: Examples of applying cross-lingual templates. Nowadays pre-trained cross-lingual language models (Conneau and Lample, 2019; Conneau et al., 2020) have become a dominant paradigm for XLU, significantly improving the performance in various XLU tasks included XNLI. Existing meth- ods (Huang et al., 2019; Chi et al., 2021a,b) usually utilize various auxiliary tasks to improve the cross- lingual transferability of a pre-trained cross-lingual language …

Content truncated to 500 words