Evaluating Approaches to Personalizing Language Models

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 2461–2469 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 2461 Evaluating Approaches to Personalizing Language Models Milton King, Paul Cook University of New Brunswick Fredericton, New Brunswick, Canada milton.king@unb.ca, paul.cook@unb.ca Abstract In this work, we consider the problem of personalizing language models, that is, building language models that are tailored to the writing style of an individual. Because training language models requires a large amount of text, and individuals do not necessarily possess a large corpus of their writing that could be used for training, approaches to personalizing language models must be able to rely on only a small amount of text from any one user. In this work, we compare three approaches to personalizing a language model that was trained on a large background corpus using a relatively small amount of text from an individual user. We evaluate these approaches using perplexity, as well as two measures based on next word prediction for smartphone soft keyboards. Our results show that when only a small amount of user-specific text is available, an approach based on priming gives the most improvement, while when larger amounts of user-specific text are available, an approach based on language model interpolation performs best. We carry out further experiments to show that these approaches to personalization outperform language model adaptation based on demographic factors. Keywords: Language modelling, Personalization, Domain adaptation 1. Introduction In this work, we consider the problem of personalizing lan- guage models, that is, building language models that are tai- lored to the writing style of an individual. Language mod- elling requires a large amount of text to sufficiently train a language model. However, a large amount of text from any one user — for example social media users such as bloggers — is not necessarily available for training a user-specific language model. To overcome this issue, some techniques treat text from other users that share demographic charac- teristics, such as age or gender, as training data. Although this has been shown to improve the performance of lan- guage models (Lynn et al., 2017), this demographic infor- mation is typically obtained via document metadata, which is not available for all document types, or for all users. This suggests a need to personalize language models while only relying on a small amount of text from the user. We propose and evaluate three different personalization techniques, while limiting the models’ access to different amounts of text from the user. These techniques are ap- plied to a language model that was trained on a large back- ground corpus of in-domain text (specifically blog posts). The three techniques include continuing to train the back- ground model on text from the user, interpolating the back- ground model with a model that was trained on the user’s text, and priming the state of the background model with text from the user. Our results show that we are able to improve the performance …

Content truncated to 500 words