AmericasNLI: Evaluating Zero-shot Natural Language Understanding of Pretrained Multilingual Models in Truly Low-resource Languages

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 6279 - 6299 May 22-27, 2022 c⃝2022 Association for Computational Linguistics AmericasNLI: Evaluating Zero-shot Natural Language Understanding of Pretrained Multilingual Models in Truly Low-resource Languages Abteen Ebrahimi♦ Manuel Mager♠ Arturo Oncevay♥ Vishrav Chaudhary§ Luis Chiruzzo△ Angela Fan∇ John E. OrtegaΩ Ricardo Ramosη Annette Riosψ Ivan Meza-Ruiz♯ Gustavo A. Giménez-Lugo♣ Elisabeth Mager♯ Graham Neubig▷◁ Alexis Palmer♦ Rolando Coto-Solano℧ Ngoc Thang Vu♠ Katharina Kann♦ ▷◁Carnegie Mellon University ℧Dartmouth College §Microsoft Turing ∇Facebook AI Research ΩNew York University △Universidad de la República, Uruguay ηUniversidad Tecnológica de Tlaxcala ♯Universidad Nacional Autónoma de México ♣Universidade Tecnológica Federal do Paraná ♦University of Colorado Boulder ♥University of Edinburgh ♠University of Stuttgart ψUniversity of Zurich Abstract Pretrained multilingual models are able to per- form cross-lingual transfer in a zero-shot set- ting, even for languages unseen during pre- training. However, prior work evaluating per- formance on unseen languages has largely been limited to low-level, syntactic tasks, and it remains unclear if zero-shot learning of high-level, semantic tasks is possible for un- seen languages. To explore this question, we present AmericasNLI, an extension of XNLI (Conneau et al., 2018) to 10 Indigenous lan- guages of the Americas. We conduct experi- ments with XLM-R, testing multiple zero-shot and translation-based approaches. Addition- ally, we explore model adaptation via contin- ued pretraining and provide an analysis of the dataset by considering hypothesis-only mod- els. We find that XLM-R’s zero-shot perfor- mance is poor for all 10 languages, with an av- erage performance of 38.48%. Continued pre- training offers improvements, with an average accuracy of 43.85%. Surprisingly, training on poorly translated data by far outperforms all other methods with an accuracy of 49.12%. 1 Introduction Pretrained multilingual models such as XLM (Lample and Conneau, 2019), multilingual BERT (mBERT; Devlin et al., 2019), and XLM-R (Con- neau et al., 2020) achieve strong cross-lingual trans- fer results for many languages and natural language processing (NLP) tasks. However, there exists a discrepancy in terms of zero-shot performance be- tween languages present in the pretraining data and those that are not: performance is generally highest for well-represented languages and decreases with less representation. Yet, even for unseen languages, performance is generally above chance, and model adaptation approaches have been shown to yield Language ISO Family Dev Test Aymara aym Aymaran 743 750 Asháninka cni Arawak 658 750 Bribri bzd Chibchan 743 750 Guaraní gn Tupi-Guaraní 743 750 Nahuatl nah Uto-Aztecan 376 738 Otomí oto Oto-Manguean 222 748 Quechua quy Quechuan 743 750 Rarámuri tar Uto-Aztecan 743 750 Shipibo-Konibo shp Panoan 743 750 Wixarika hch Uto-Aztecan 743 750 Table 1: The languages in AmericasNLI, along with their ISO codes, language families, and dataset sizes. further improvements (Muller et al., 2020; Pfeiffer et al., 2020a,b; Wang et al., 2020). Importantly, however, there are currently no datasets for high-level, semantic tasks which fo- cus solely on low-resource languages. As these languages are most likely to be unseen to com- monly used pretrained models, practically …

Content truncated to 500 words