XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10215–10245 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 10215 XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation Sebastian Ruder1, Noah Constant2, Jan Botha2, Aditya Siddhant2, Orhan Firat2, Jinlan Fu3, Pengfei Liu4, Junjie Hu4, Dan Garrette2, Graham Neubig4, Melvin Johnson2 1DeepMind 2Google Research 3Fudan University 4Carnegie Mellon University Abstract Machine learning has brought striking ad- vances in multilingual natural language pro- cessing capabilities over the past year. For ex- ample, the latest techniques have improved the state-of-the-art performance on the XTREME multilingual benchmark by more than 13 points. While a sizeable gap to human- level performance remains, improvements have been easier to achieve in some tasks than in others. This paper analyzes the cur- rent state of cross-lingual transfer learning and summarizes some lessons learned. In order to catalyze meaningful progress, we extend XTREME to XTREME-R, which con- sists of an improved set of ten natural lan- guage understanding tasks, including challeng- ing language-agnostic retrieval tasks, and cov- ers 50 typologically diverse languages. In ad- dition, we provide a massively multilingual di- agnostic suite (MULTICHECKLIST) and fine- grained multi-dataset evaluation capabilities through an interactive public leaderboard to gain a better understanding of such models. 1 Introduction Most research in natural language processing (NLP) to date has focused on developing methods that work well for English and a small set of other high-resource languages (Joshi et al., 2020). In contrast, methods for other languages can be vastly more beneficial as they enable access to language technology for more than three billion speakers of low-resource languages and prevent the NLP com- munity from overfitting to English. Motivated by these benefits, the area of multilingual NLP has attracted increasing interest recently. However, evaluating multilingual models is chal- lenging as it requires assessing performance on a wide range of typologically distinct languages in the face of limited heterogeneous data sources. Re- cently large-scale benchmarks such as XTREME (Hu et al., 2020) and XGLUE (Liang et al., 2020) XTREME XTREME-R # of languages 40 50 # of tasks 9 − 2 + 3 = 10 Task categories Classification, structured prediction, QA, retrieval +language-agnostic retrieval Analysis tools — MULTICHECKLIST, Explainaboard Leaderboard Static Interactive, +metadata Table 1: Overview of XTREME and XTREME-R. have been introduced, consolidating existing mul- tilingual tasks and covering tens of languages. When XTREME was released, the gap between the best-performing baseline, XLM-R Large (Conneau et al., 2020), and human-level performance was roughly 25. This has since shrunk to less than 12 points, a much smaller but still substantial gap com- pared to the difference from human-level perfor- mance observed in English transfer learning (Wang et al., 2019a), which has recently been closed en- tirely on some evaluation suites (He et al., 2021). In order to examine the nature of this progress, we first perform an analysis of state-of-the-art mul- tilingual models on XTREME. We observe that progress has not been uniform, but concentrated on cross-lingual retrieval tasks where fine-tuning on …

Content truncated to 500 words