Are “Undocumented Workers” the Same as “Illegal Aliens”? Disentangling Denotation and Connotation in Vector Spaces

Are “Undocumented Workers” the Same as “Illegal Aliens”? Disentangling Denotation and Connotation in Vector Spaces Albert Webson1,2, Zhizhong Chen3, Carsten Eickhoff1, and Ellie Pavlick1 {albert webson, zhizhong chen, carsten, ellie pavlick}@brown.edu 1Department of Computer Science, Brown University 2Department of Philosophy, Brown University 3Department of Physics, Brown University Abstract In politics, neologisms are frequently invented for partisan objectives. For example, “undoc- umented workers” and “illegal aliens” refer to the same group of people (i.e., they have the same denotation), but they carry clearly differ- ent connotations. Examples like these have traditionally posed a challenge to reference- based semantic theories and led to increasing acceptance of alternative theories (e.g., Two- Factor Semantics) among philosophers and cognitive scientists. In NLP, however, pop- ular pretrained models encode both denota- tion and connotation as one entangled repre- sentation. In this study, we propose an ad- versarial neural network that decomposes a pretrained representation as independent deno- tation and connotation representations. For intrinsic interpretability, we show that words with the same denotation but different conno- tations (e.g., “immigrants” vs. “aliens”, “estate tax” vs. “death tax”) move closer to each other in denotation space while moving further apart in connotation space. For extrinsic application, we train an information retrieval system with our disentangled representations and show that the denotation vectors improve the viewpoint diversity of document rankings. 1 Introduction Language carries information through both deno- tation and connotation. For example, a reporter writing an article about the leftmost wing of the Democratic party can choose to refer to the group as “progressives” or as “radicals”. The word choice does not change the individuals referred to, but it does communicate significantly different senti- ments about the policy positions discussed. This type of linguistic nuance presents a significant challenge for natural language processing systems, most of which fundamentally assume words to have similar meanings if they are surrounded in similar single payer insurance program affordable health government run taxpayer funded horror stories totalitarian stimulus bill spending cuts obama policies freedomworks trillion dollar ryan budget wealthiest americans Figure 1: Nearest neighbors of government-run health- care (triangles) and economic stimulus (circles). Note that words cluster as strongly by policy denotation (shapes) as by partisan connotation (colors); namely, pretrained representations conflate denotation with con- notation. Plotted by t-SNE with perplexity = 10. word contexts. Such assumption risks confusing differences in connotation for differences in deno- tation or vice versa. For example, using a common skip-gram model (Mikolov et al., 2013) trained on a news corpus (described in §3.2), Figure 1 shows nearest neighbors of “government-run healthcare” and “economic stimulus”. The resulting t-SNE clusters are influenced as much by policy deno- tation (shapes) as they are by partisan connota- tion (colors1). Using these entangled representa- tions in applications such as information retrieval could have pernicious consequences such as rein- forcing ideological echo chambers and political polarization. For example, a right-leaning query like “taxpayer-funded healthcare” could make one equally (if not more) likely to see articles about “totalitarian” and “horror stories” than about “af- …

Content truncated to 500 words