GeoMLAMA: Geo-Diverse Commonsense Probing on Multilingual Pre-Trained Language Models

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 2039 - 2055 December 7-11, 2022 ©2022 Association for Computational Linguistics GEOMLAMA: Geo-Diverse Commonsense Probing on Multilingual Pre-Trained Language Models Da Yin Hritik Bansal Masoud Monajatipoor Liunian Harold Li Kai-Wei Chang Computer Science Department, University of California, Los Angeles {da.yin,hbansal,liunian.harold.li,kwchang}@cs.ucla.edu, monajati@ucla.edu Abstract Recent work has shown that Pre-trained Lan- guage Models (PLMs) store the relational knowledge learned from data and utilize it for performing downstream tasks. However, com- monsense knowledge across different regions may vary. For instance, the color of bridal dress is white in American weddings whereas it is red in Chinese weddings. In this paper, we introduce a benchmark dataset, Geo-diverse Commonsense Multilingual Language Models Analysis (GEOMLAMA), for probing the di- versity of the relational knowledge in multi- lingual PLMs. GEOMLAMA contains 3,125 prompts in English, Chinese, Hindi, Persian, and Swahili, with a wide coverage of concepts shared by people from American, Chinese, In- dian, Iranian and Kenyan cultures. We bench- mark 11 standard multilingual PLMs on GE- OMLAMA. Interestingly, we find that 1) larger multilingual PLMs variants do not necessar- ily store geo-diverse concepts better than its smaller variant; 2) multilingual PLMs are not intrinsically biased towards knowledge from the Western countries (the United States); 3) the native language of a country may not be the best language to probe its knowledge and 4) a lan- guage may better probe knowledge about a non- native country than its native country. Code and data are released at https://github. com/WadeYin9712/GeoMLAMA. 1 Introduction Pre-trained Language Models (PLMs) (Peters et al., 2018; Radford et al., 2019; Devlin et al., 2019; Brown et al., 2020) are increasingly used in various Natural Language Processing (NLP) applications. Pre-trained on large-scale text corpora, they are shown to store relational knowledge (Petroni et al., 2019; Jiang et al., 2020b; Kassner et al., 2021), e.g., commonsense knowledge (Zhou et al., 2020; Lin et al., 2020; Nguyen et al., 2021; Zhou et al., 2021). They have been used to construct knowledge bases while requiring limited human effort for rule cre- In traditional [X] weddings, the color of wedding dress is usually [MASK]. पारंप�रक [X] शािदयों म� दु�न की पोशाक का रंग आमतौर पर [MASK] होता है। Kwenye harusi za kitamaduni nchini [X], rangi ya mavazi ya bibi harusi huwa [MASK]. ... [X] (Country name) [MASK] American white Chinese red Indian red Iranian white Kenyan white Color of wedding dress [X] (Country name) [MASK] अमे�रकी सफे द (white) चीनी लाल (red) भारतीय लाल (red) फ़ारसी सफे द (white) के �ाी सफे द (white) ... EN HI SW Figure 1: Examples of prompts and gold answers in GEOMLAMA. For each concept (e.g., color of wedding dress), there are multiple masked multilingual prompts (English, Hindi, Swahili, etc.) with specified country in- formation [X] querying geo-diverse knowledge about the concept. We test multilingual PLMs by examining the extent to which masked word predictions align with the gold answers in [MASK] columns. ation and validation (Bosselut et al., 2019; Zhou …

Content truncated to 500 words