XCOPA: A Multilingual Dataset for Causal Commonsense Reasoning

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 2362–2376, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 2362 XCOPA: A Multilingual Dataset for Causal Commonsense Reasoning Edoardo M. Ponti1∗, Goran Glavaˇs2∗, Olga Majewska1, Qianchu Liu1, Ivan Vuli´c1, Anna Korhonen1 1Language Technology Lab, TAL, University of Cambridge, UK 2 Data and Web Science Group, University of Mannheim, Germany 1{ep490,om304,ql261,iv250,alk23}@cam.ac.uk 2goran@informatik.uni-mannheim.de Abstract In order to simulate human language ca- pacity, natural language processing systems must be able to reason about the dynamics of everyday situations, including their pos- sible causes and effects. Moreover, they should be able to generalise the acquired world knowledge to new languages, modulo cultural differences. Advances in machine reasoning and cross-lingual transfer depend on the availability of challenging evaluation benchmarks. Motivated by both demands, we introduce Cross-lingual Choice of Plausi- ble Alternatives (XCOPA), a typologically di- verse multilingual dataset for causal common- sense reasoning in 11 languages, which in- cludes resource-poor languages like Eastern Apur´ımac Quechua and Haitian Creole. We evaluate a range of state-of-the-art models on this novel dataset, revealing that the perfor- mance of current methods based on multilin- gual pretraining and zero-shot fine-tuning falls short compared to translation-based transfer. Finally, we propose strategies to adapt multi- lingual models to out-of-sample resource-lean languages where only a small corpus or a bilin- gual dictionary is available, and report sub- stantial improvements over the random base- line. The XCOPA dataset is freely available at github.com/cambridgeltl/xcopa. 1 Introduction Commonsense reasoning is a critical component of any natural language understanding system (Davis and Marcus, 2015). Contrary to textual entailment, commonsense reasoning must bridge between premises and possible hypotheses with world knowledge that is not explicit in text (Singer et al., 1992). Such world knowledge encompasses, among other aspects: temporal and spatial relations, causality, laws of nature, social conventions, polite- ness, emotional responses, and multiple modalities. ∗Equal contribution. Ultimately, it shapes the individuals’ expectations about typical situations (Shoham, 1990).1 A seminal work on the quantitative evaluation of commonsense reasoning is the Choice Of Plausi- ble Alternatives dataset (COPA; Roemmele et al., 2011), which focuses on cause–effect relationships. In recent years, more datasets have been dedicated to other facets of world knowledge (Sakaguchi et al., 2020; Bisk et al., 2020; Bhagavatula et al., 2020; Rashkin et al., 2018; Sap et al., 2019, inter alia). Unfortunately, the extensive efforts related to this thread of research have so far been limited only to the English language.2 Such a narrow scope not only curbs the development of natural language un- derstanding tools in other languages (Bender, 2011; Ponti et al., 2019a), but also exacerbates the Anglo- centric bias in modeling commonsense reasoning. In fact, the expectations about typical situations do vary across cultures (Thomas, 1983). Datasets that cover multiple languages for other natural understanding tasks, such as language in- ference (Conneau et al., 2018), question answering (Lewis et al., 2020; Artetxe et al., 2020a; Clark et al., 2020), and paraphrase identification (Yang et al., 2019b) have received increasing attention. In …

Content truncated to 500 words