Predicting Responses to Psychological Questionnaires from Participants’ Social Media Posts and Question Text Embeddings

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1512–1524 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 1512 Predicting Responses to Psychological Questionnaires from Participants’ Social Media Posts and Question Text Embeddings Huy Vu1, Suhaib Abdurahman2, Sudeep Bhatia3, Lyle Ungar3 1Stony Brook University, 2Free University of Berlin, 3University of Pennsylvania hvu@cs.stonybrook.edu, suhaib.abdurahman@gmail.com, bhatiasu@sas.upenn.edu, ungar@cis.upenn.edu Abstract Psychologists routinely assess people’s emo- tions and traits, such as their personality, by collecting their responses to survey question- naires. Such assessments can be costly in terms of both time and money, and often lack generalizability, as existing data cannot be used to predict responses for new survey ques- tions or participants. In this study, we propose a method for predicting a participant’s ques- tionnaire response using their social media texts and the text of the survey question they are asked. Specifically, we use Natural Lan- guage Processing (NLP) tools such as BERT embeddings to represent both participants (via the text they write) and survey questions as embeddings vectors, allowing us to predict responses for out-of-sample participants and questions. Our novel approach can be used by researchers to integrate new participants or new questions into psychological studies with- out the constraint of costly data collection, fa- cilitating novel practical applications and fur- thering the development of psychological the- ory. Finally, as a side contribution, the success of our model also suggests a new approach to study survey questions using NLP tools such as text embeddings rather than response data used in traditional methods. 1 Introduction Psychologists conduct personality research in order to understand what aspects and factors consistently distinguish people from each other on an individual level. This is relevant because personality influences important life outcomes such as occupational and educational success and even physical and mental health (Judge et al., 1999; Roberts et al., 2007). Traditionally, psychologists measure personality through questionnaires, by having participants read Figure 1: Overview of proposed task: analyzing users’ social media text and questions text to predict re- sponses. and answer questions on a rating scale, for instance from ”strongly disagree” to ”strongly agree”. How- ever, acquiring questionnaire data in psychological research is often a tedious and costly process, as study participants must be recruited and motivated to complete questionnaires. This problem is partic- ularly pronounced for longer surveys, which suffer from low completion rates and careless responses due to low participant motivation (Niessen et al., 2016; Van de Mortel et al., 2008; Raghunathan and Grizzle, 1995; Champion and Sear, 1969). There- fore, the ability to predict questionnaire responses would be of great use to researchers. The main contribution of this paper is to address this issue. We propose a system that uses the par- ticipants’ social media texts and the question texts to predict the participants’ responses. The system extracts BERT embeddings from the two input com- ponents and then trains a predictive model. After training, we can predict the responses for every new participant, new question or both, only requiring 1513 the participants’ social …

Content truncated to 500 words