Richer Countries and Richer Representations

Findings of the Association for Computational Linguistics: ACL 2022, pages 2074 - 2085 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Richer Countries and Richer Representations Kaitlyn Zhou Stanford University katezhou@stanford.edu Kawin Ethayarajh Stanford University kawin@stanford.edu Dan Jurafsky Stanford University jurafsky@stanford.edu Abstract We examine whether some countries are more richly represented in embedding space than others. We find that countries whose names occur with low frequency in training corpora are more likely to be tokenized into subwords, are less semantically distinct in embedding space, and are less likely to be correctly pre- dicted: e.g., Ghana (the correct answer and in- vocabulary) is not predicted for, “The country producing the most cocoa is [MASK].”. Al- though these performance discrepancies and representational harms are due to frequency, we find that frequency is highly correlated with a country’s GDP; thus perpetuating his- toric power and wealth inequalities. We an- alyze the effectiveness of mitigation strate- gies; recommend that researchers report train- ing word frequencies; and recommend future work for the community to define and design representational guarantees. 1 Introduction How similar are the words “Brooklyn” and “Queens”? To a New Yorker, they evoke two very different places, cultures, and cuisines, but to a Seattleite, they are quite similar, both being bor- oughs of New York City.1 Our perception of en- tities such as cities or countries is conditioned on our backgrounds. Here, we ask if language models are also susceptible to representational biases. We suggest three criteria to characterize the quality of representations for particular entities or groups: consistency, distinctiveness, and recogniz- ability. For consistency, are all entities of a certain type (such as all country names) represented with the same number of tokens in the lexicon? For dis- tinctiveness, are entities of the same category seen as equally distinct in representational space? For recognizability, are models capable of generating 1https://en.wikipedia.org/wiki/View_ of_the_World_from_9th_Avenue. all entities of a certain type in response to ques- tions? And are the differences between entities confounded across lines of historical inequity (like wealth of countries)? Focusing on BERT (bert-base-cased2) representations (Devlin et al., 2019), we find that names of countries that appear less fre- quently in training data are less likely to be in- vocabulary, are less semantically distinct from other countries, and are less frequently pre- dicted in the masked language modeling (MLM) task. Disappointingly, we find similar behavior in bert-base-multilingual-cased and roberta-base. We identify these differences as intrinsic representational harms where low fre- quency countries are more likely to be conflated with one another and their existence less recog- nized. A more troubling result is that training data fre- quency is highly correlated with the gross domestic product of a country (GDP) (Pearson’s r = 0.82). Our training data and thus the representation of en- tities through our language models encodes wealth and power disparities and perpetuates representa- tional harms. Given these significant differences in representation, what could it look like to impose a minimum quality of representation for signifi- cant entities? We recommend that the community …

Content truncated to 500 words