Common Sense Beyond English: Evaluating and Improving Multilingual Language Models for Commonsense Reasoning

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 1274–1287 August 1–6, 2021. ©2021 Association for Computational Linguistics 1274 Common Sense Beyond English: Evaluating and Improving Multilingual Language Models for Commonsense Reasoning Bill Yuchen Lin Seyeon Lee Xiaoyang Qiao Xiang Ren {yuchen.lin, seyeonle, xiaoyanq, xiangren}@usc.edu Department of Computer Science and Information Sciences Institute, University of Southern California Abstract Commonsense reasoning research has so far been limited to English. We aim to evalu- ate and improve popular multilingual language models (ML-LMs) to help advance common- sense reasoning (CSR) beyond English. We collect the Mickey corpus, consisting of 561k sentences in 11 different languages, which can be used for analyzing and improving ML- LMs. We propose Mickey Probe, a language- agnostic probing task for fairly evaluating the common sense of popular ML-LMs across dif- ferent languages. In addition, we also create two new datasets, X-CSQA and X-CODAH, by translating their English versions to 15 other languages, so that we can evaluate pop- ular ML-LMs for cross-lingual commonsense reasoning. To improve the performance be- yond English, we propose a simple yet effec- tive method — multilingual contrastive pre- training (MCP). It significantly enhances sen- tence representations, yielding a large perfor- mance gain on both benchmarks (e.g., +2.7% accuracy for X-CSQA over XLM-RL)1. 1 Introduction Understanding natural language relies heavily on commonsense reasoning (CSR), which is the pro- cess of making inferences with commonsense knowledge. Commonsense knowledge is the set of general facts that reflect our natural understanding of the physical world and human behavior, which are usually seen as an implicit background when people communicate with each other using lan- guages. It is thus of vital importance to evalu- ate and improve the commonsense reasoning ca- pability of language models (LMs), towards build- ing general natural language understanding (NLU) systems (Davis and Marcus, 2015). 1We release our code and data at the project website: https://inklab.usc.edu/XCSR/. Birds have [mask] . LAMA Probe Where do adults usually use glue sticks? A) school B) drawer C) office CommonsenseQA SWAG/CODAH The chef drops the piece of shrimp in the fryer. → A) The chef chops the pan. B) The chef watches it sizzle. C) The chef likes fried chicken. English LM Multilingual LM EN AR ZH FR HI IT DE ES JA PL NL PT UR VI RU SW wing C B Figure 1: Commonsense reasoning is well-studied with benchmarks and LMs in English. Can we advance commonsense reasoning beyond English? Many recent benchmark datasets and probing methods have been proposed to evaluate ma- chine common sense. As shown in Figure 1, the LAMA probe (Petroni et al., 2019) is for an- alyzing LMs’ zero-shot commonsense recalling ability; CommonsenseQA (CSQA) (Talmor et al., 2019) is instead a multiple-choice QA task that needs fine-tuning; CODAH (Chen et al., 2019) and SWAG (Zellers et al., 2018) focus on the abil- ity to complete the most plausible scenes. How- ever, all these works have been limited …

Content truncated to 500 words