IndicXNLI: Evaluating Multilingual Inference for Indian Languages

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10994 - 11006 December 7-11, 2022 ©2022 Association for Computational Linguistics INDICXNLI: Evaluating Multilingual Inference for Indian Languages Divyanshu Aggarwal1∗, Vivek Gupta2*†, Anoop Kunchukuttan3, 4 1Delhi Technological University; 2University of Utah; 3Microsoft IDC; 4AI4Bharat divyanshuggrwl@gmail.com ; vgupta@cs.utah.edu ; ankunchu@microsoft.com Abstract While Indic NLP has made rapid advances re- cently in terms of the availability of corpora and pre-trained models, benchmark datasets on standard NLU tasks are limited. To this end, we introduce INDICXNLI, an NLI dataset for 11 Indic languages. It has been created by high-quality machine translation of the orig- inal English XNLI dataset and our analysis attests to the quality of INDICXNLI. By fine- tuning different pre-trained LMs on this IN- DICXNLI, we analyze various cross-lingual transfer techniques with respect to the im- pact of the choice of language models, lan- guages, multi-linguality, mix-language input, etc. These experiments provide us with use- ful insights into the behaviour of pre-trained models for a diverse set of languages. 1 Introduction Natural Language Inference (NLI) is a well-studied NLP task (Dagan et al., 2013) that assesses if a premise entails, negates, or is neutral towards the hypothesis statement. The task is well suited for evaluating semantic representations of state-of- the-art transformers (Vaswani et al., 2017) mod- els such as BERT (Devlin et al., 2019; Radford and Narasimhan, 2018). Two large scale datasets, such as SNLI (Bowman et al., 2015) and MultiNLI (Williams et al., 2018), has recently been developed to enhanced the relevance of the NLI task. With the availability of multi-lingual pre-trained language models such as mBERT (Devlin et al., 2019), and XLM-RoBERTa (Conneau et al., 2020a) promising cross-lingual transfer and universal mod- els, multi-lingual NLP has recently gained a lot of attention. However, most languages have a scarcity of datasets resources. Some multi-lingual datasets have attempted to fill this gap, including XNLI (Conneau et al., 2018) for NLI, XQUAD ∗Equal Contribution †Corresponding Author (Dumitrescu et al., 2021), MLQA (Lewis et al., 2020) for question answering, and PAWS-X for paraphrase identification (Yang et al., 2019). In many practical circumstances, training sets for non- English languages are unavailable, hence cross- lingual zero-shot evaluation benchmarks such as XTREME (Hu et al., 2020a), XTREME-R (Ruder et al., 2021), and XGLUE (Liang et al., 2020) have been suggested to use these datasets. However, NLI datasets are not available for ma- jor Indic languages. The only exceptions are the test/validation sets in the XNLI (hi and ur), Tax- iXNLI (hi) (K et al., 2021) and MIDAS-NLI (Up- pal et al., 2020) datasets. Furthermore, because MIDAS-NLI is based on sentiment data recasting, hypotheses are not linguistically diverse and span limited reasoning. In this work, we address this gap by introducing INDICXNLI, an NLI dataset for Indic languages. INDICXNLI consists of En- glish XNLI data translated into eleven Indic lan- guages. We use INDICXNLI to evaluate Indic- specific models (trained only on Indic and English languages) such as IndicBERT (Kakwani et al., 2020) and …

Content truncated to 500 words