Human Language Modeling

Findings of the Association for Computational Linguistics: ACL 2022, pages 622 - 636 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Human Language Modeling Nikita Soni, Matthew Matero, Niranjan Balasubramanian, and H. Andrew Schwartz Department of Computer Science, Stony Brook University {nisoni, mmatero, niranjan, has}@cs.stonybrook.edu Abstract Natural language is generated by people, yet traditional language modeling views words or documents as if generated independently. Here, we propose human language modeling (HuLM), a hierarchical extension to the lan- guage modeling problem whereby a human- level exists to connect sequences of documents (e.g. social media messages) and capture the notion that human language is moderated by changing human states. We introduce, HaRT, a large-scale transformer model for the HULM task, pre-trained on approximately 100,000 so- cial media users, and demonstrate it’s effec- tiveness in terms of both language modeling (perplexity) for social media and fine-tuning for 4 downstream tasks spanning document- and user-levels: stance detection, sentiment classification, age estimation, and personality assessment.1 Results on all tasks meet or sur- pass the current state-of-the-art. 1 Introduction Language use, like any human behavior, is moder- ated by underlying human states of being (Mehl and Pennebaker, 2003; Fleeson, 2001). Indeed, different ways of incorporating human informa- tion into NLP models have recently been shown to improve accuracy on many NLP tasks (Hovy, 2015; Lynn et al., 2017; Huang and Paul, 2019; Hovy and Yang, 2021). At the same time, while language modeling has proven itself fundamental to NLP, it is typically absent the notion of a human producing the natural language. From a statistical modeling perspective, this ab- sence of human state can be seen as an instance of the ecological fallacy – the treatment of mul- tiple observations (i.e. text sequences) from the same source (i.e. human) as independent (Pianta- dosi et al., 1988; Steel and Holt, 1996). 1Code and pre-trained models available at: https://github.com/humanlab/HaRT. To address this, we introduce the task of human language modeling (HULM), which induces de- pendence among text sequences via the notion of a human state in which the text was generated. In particular, we formulate HULM as the task of es- timating the probability of a sequence of tokens, wt,1∶i, while conditioning on a higher order state (U1∶t−1) derived from the tokens of other docu- ments written by the same individual. Its key ob- jective is: Pr(wt,i∣wt,1∶i−1, U1∶t−1) where t indexes a particular sequence of tempo- rally ordered utterances (e.g. a document or so- cial media post), and U1∶t−1 represents the human state just before the current sequence, t. In one ex- treme, U1∶t−1 could model all previous tokens in all previous documents by the person. In the oppo- site extreme, U1∶t−1 can be the same for all users and for values of t reducing to standard language modeling: Pr(wi∣w1∶i−1).2 Thus, HULM-based models without history can be used where tradi- tional LMs are applied (and may even perform bet- ter). HULM brings together ideas from human fac- tor inclusion/adaptation (Hovy, 2015; Lynn et al., 2017; Hovy and Yang, …

Content truncated to 500 words