Leveraging Similar Users for Personalized Language Modeling with Limited Data

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 1742 - 1752 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Leveraging Similar Users for Personalized Language Modeling with Limited Data Charles Welch ∗ and Chenxi Gu ∗ and Jonathan K. Kummerfeld and Verónica Pérez-Rosas and Rada Mihalcea Conversational AI and Social Analytics (CAISA) Lab Department of Mathematics and Computer Science, University of Marburg Language and Information Technologies Lab (LIT) Department of Computer Science and Engineering, University of Michigan Abstract Personalized language models are designed and trained to capture language patterns specific to individual users. This makes them more accurate at predicting what a user will write. However, when a new user joins a platform and not enough text is available, it is harder to build effective personalized language models. We propose a solution for this problem, using a model trained on users that are similar to a new user. In this paper, we explore strategies for finding the similarity between new users and existing ones and methods for using the data from existing users who are a good match. We further explore the trade-off between available data for new users and how well their language can be modeled. 1 Introduction Recent work has suggested that there are several benefits to personalized models in natural language processing (NLP) over one-size-fits-all solutions: they are more accurate for individual users; they help us understand communities better; and they focus the attention of our evaluations on the end- user (Flek, 2020). Generation tasks in particular benefit from a personalized approach, for example, Dudy et al. (2021) argue that user intention is more often difficult to recover from the context alone. We study personalization in language modeling, a core task in NLP. Direct applications of language models (LM) include predictive text, authorship attribution, and dialog systems used to model the style of an individual or profession (e.g., thera- pist, counselor). LMs are increasingly used as the backbone of models for a range of tasks in NLP, increasing the potential impact of personalization even further (Brown et al., 2020). The standard non-personalized approach is to use pretrained models trained on a large volume *Authors contributed equally and work was performed while at the University of Michigan. of data written by many people. This approach does not take into account the differences between individuals and their language patterns. Given the same context, different people may act or write dif- ferently, but these general models cannot produce that type of variation. Approaches like fine-tuning can be used to tailor a pretrained model to an indi- vidual, but perform well only when enough data is available, which is often not the case. Previous work on personalized and demographic word embeddings has seen successful application in downstream tasks. Garimella et al. (2017) look at location and gender and how they affect asso- ciations with words like “health” and many other stimulus words like “stack”– does it make you think of books or …

Content truncated to 500 words