When do Word Embeddings Accurately Reflect Surveys on our Beliefs About People?

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4392–4415 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 4392 When do Word Embeddings Accurately Reflect Surveys on our Beliefs About People? Kenneth Joseph Computer Science and Engineering University at Buffalo Buffalo, NY, 14226 kjoseph@buffalo.edu Jonathan H. Morgan University of Applied Sciences Fachhochschule Potsdam Potsdam, Germany morgan@fh-potsdam.de Abstract Social biases are encoded in word embeddings. This presents a unique opportunity to study so- ciety historically and at scale, and a unique danger when embeddings are used in down- stream applications. Here, we investigate the extent to which publicly-available word em- beddings accurately reflect beliefs about cer- tain kinds of people as measured via traditional survey methods. We find that biases found in word embeddings do, on average, closely mir- ror survey data across seventeen dimensions of social meaning. However, we also find that biases in embeddings are much more re- flective of survey data for some dimensions of meaning (e.g. gender) than others (e.g. race), and that we can be highly confident that embedding-based measures reflect survey data only for the most salient biases. 1 Introduction In April of 2015, protests erupted over the mur- der of Freddie Gray, Jr. Questions about what to call those protesting quickly became the focus of a national debate. In an interview on CNN with Erin Burnett,1 Baltimore City Councilman Carl Stokes admonished then-President Barack Obama and then-Mayor Stephanie Rawlings-Blake for us- ing the word thugs to refer to the protesters. Bur- nett challenged Stokes’ admonition, claiming the protesters were indeed thugs because “They know it’s wrong to steal and burn.” Stokes responded by stating the protesters were “...children who have been set aside [and] marginalized.” The argument between Burnett and Stokes is over the way we label people, the meanings of those labels, and the impacts these meanings can have. Councilman Stokes wants to avoid using the label “thug” because of its established, negative 1http://nymag.com/intelligencer/2015/04/carl-stokes-to- cnn-thug-is-racially-charged.html | | | | || | || || | ||| | || | | | |||| ||| | || ||| |||| |||| || | | | | | || || | | |||| | | | | | | || ||||| || | | ||||||||| |||||| | | || || | | | | || | | | | | | ||||| || |||| | | || | || | | ||||| | | | | | | | | | | | | || | | | || | | || || | | || ||| | | | | | ||| |||| |||| || | | | || | | | || | | ||| | | | || || || | || ||| | | || ||| | || | || | | | ||| | | | | | ||| || || | thug child judge secretary Potency (weak−−strong) Evaluation (bad−−good) Gender (man−−woman) Black (not black−−black) Belief Dimension Figure 1: Beliefs (x-axis) about four identities (sepa- rate plots) …

Content truncated to 500 words