Exploring the Value of Personalized Word Embeddings

Proceedings of the 28th International Conference on Computational Linguistics, pages 6856–6862 Barcelona, Spain (Online), December 8-13, 2020 6856 Exploring the Value of Personalized Word Embeddings Charles Welch, Jonathan K. Kummerfeld, Ver´onica P´erez-Rosas and Rada Mihalcea Computer Science & Engineering University of Michigan {cfwelch,jkummerf,vrncapr,mihalcea}@umich.edu Abstract In this paper, we introduce personalized word embeddings, and examine their value for language modeling. We compare the performance of our proposed prediction model when using personal- ized versus generic word representations, and study how these representations can be leveraged for improved performance. We provide insight into what types of words can be more accurately predicted when building personalized models. Our results show that a subset of words belong- ing to specific psycholinguistic categories tend to vary more in their representations across users and that combining generic and personalized word embeddings yields the best performance, with a 4.7% relative reduction in perplexity. Additionally, we show that a language model using personalized word embeddings can be effectively used for authorship attribution. 1 Introduction Word embeddings have become ubiquitous in natural language processing applications. Usually, em- beddings are trained from a large corpus of news or web data that contains writing from many sources and authors (Mikolov et al., 2013; Pennington et al., 2014). These embeddings capture syntactic and semantic properties of the language of all authors who contributed to this corpus. Multi-source corpora provide large volumes of data, but they may not lead to the ideal representations for individuals. For instance, the word “hometown” may have a different representation for different individuals. For some, it may relate to words such as “hills,” “trees,” and “family,” whereas for others may be more strongly connected to “ocean,” “beach,” and “friends.” These personalized representations differ among individuals, and also differ from a more generic representation that often tends to capture words that are semantically related at concept level, such as “city,” “town,” or “place.” In this paper, we present the idea of personalized word embeddings. We explore differences in person- alized word representations using a corpus of English Reddit posts that contains a large number of posts per author. We use the embeddings to initialize a language model and show that personalization leads to better results than generic embeddings. One potential application of this work is personalized text generation for auto-completion to speed up text entry. Another application is dialog systems that follow the speaking style of certain professionals (e.g., counselors, advisors). Finally, personalized word repre- sentations could particularly help users with atypical writing styles that are not currently well served by models trained to suit the majority. 2 Related Work Prior work has considered user embeddings, where one vector is learned for each user in the data (we learn a set of vectors per user, one for each word in the vocabulary). User embeddings have been used for dialog generation (Li et al., 2016), query auto-completion (Jaech and Ostendorf, 2018), authorship attribution (Ebrahimi and Dou, 2016), and sarcasm detection (Kolchinski and Potts, 2018). Amer et al. (2016) learn a …

Content truncated to 500 words