Compositional Demographic Word Embeddings

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 4076–4089, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 4076 Compositional Demographic Word Embeddings Charles Welch, Jonathan K. Kummerfeld, Ver´onica P´erez-Rosas, and Rada Mihalcea Computer Science & Engineering University of Michigan {cfwelch,jkummerf,vrncapr,mihalcea}@umich.edu Abstract Word embeddings are usually derived from corpora containing text from many individu- als, thus leading to general purpose representa- tions rather than individually personalized rep- resentations. While personalized embeddings can be useful to improve language model per- formance and other language processing tasks, they can only be computed for people with a large amount of longitudinal data, which is not the case for new users. We propose a new form of personalized word embeddings that use demographic-specific word representations de- rived compositionally from full or partial de- mographic information for a user (i.e., gen- der, age, location, religion). We show that the resulting demographic-aware word representa- tions outperform generic word representations on two tasks for English: language modeling and word associations. We further explore the trade-off between the number of available at- tributes and their relative effectiveness and dis- cuss the ethical implications of using them. 1 Introduction Word embeddings are used in many natural lan- guage processing tasks as a way of representing language. Embeddings can be efficiently trained on large corpora using methods like word2vec or GloVe (Mikolov et al., 2013; Pennington et al., 2014), which learn one vector per word. These embeddings capture syntactic and semantic prop- erties of the language of all individuals who con- tributed to the corpus. However, they are unable to account for user-specific word preferences (e.g., using the same word in different ways across dif- ferent contexts), particularly for individuals whose usage deviates from the majority. These individ- ual preferences are reflected in the word’s nearest neighbors. As an example, Table 1 shows the way two users use the word “health” and the word’s five nearest neighbors in their respective personalized embedding spaces. The word is used in similar User Example Use Nearest Neighbors A doctors think this is bad for her health ... preventative, insurance, re- form, medical, education B it is usually bad for your health ... professional, mental, con- duct, experiences, online All N/A medical, preventative, in- surance, safety, healthcare Table 1: Nearest neighbors of the word “health” for two different users in personalized and a generic em- bedding space. contexts, where contextual embeddings may give similar representations, but it has different salient meanings in the personal space of each user. User A tends to talk more about preventative care and insurance, while user B tends to talk about people’s experiences affecting their mental health. The typical approach in natural language pro- cessing (NLP) is to use one-size-fits-all language representations, which do not account for variation between people. This may not matter for people whose language style is well represented in the data, but could lead to worse support for others (Pavalanathan and Eisenstein, 2015; May et al., 2019; Kurita et al., 2019). …

Content truncated to 500 words