Constructing a Psychometric Testbed for Fair Natural Language Processing

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3748–3758 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 3748 Constructing a Psychometric Testbed for Fair Natural Language Processing Ahmed Abbasi,1,2∗David Dobolyi,1,2 John P. Lalor,1,2 Richard Netemeyer,3 Kendall Smith,1,2 Yi Yang4 1 Human-centered Analytics Lab, University of Notre Dame 2 Department of IT, Analytics, and Operations, University of Notre Dame 3 Center for Data Analytics, University of Virginia 4 Department of Information Systems and Operations Management, HKUST {aabbasi, ddobolyi, john.lalor, ksmith77}@nd.edu, rgn3p@virginia.edu, imyiyang@ust.hk Abstract Psychometric measures of ability, attitudes, perceptions, and beliefs are crucial for un- derstanding user behavior in various contexts including health, security, e-commerce, and finance. Traditionally, psychometric dimen- sions have been measured and collected us- ing survey-based methods. Inferring such con- structs from user-generated text could allow timely, unobtrusive collection and analysis. In this work we construct a corpus for psycho- metric natural language processing (NLP) re- lated to important dimensions such as trust, anxiety, numeracy, and literacy, in the health domain. We discuss our multi-step process to align user text with their survey-based re- sponse items and provide an overview of the resulting testbed, which encompasses survey- based psychometric measures and accompa- nying user-generated text from 8,502 respon- dents. Our testbed also encompasses self- reported demographic information, including race, sex, age, income, and education, allow- ing for measuring bias and benchmarking fair- ness of text classification methods. We report preliminary results on use of the text to pre- dict/categorize users’ survey response labels and on the fairness of these models. We also discuss the important implications of our work and resulting testbed for future NLP research on psychometrics and fairness. 1 Introduction Psychometrics is the field of study concerned with the measurement of individuals’ knowledge, abil- ities, attitudes, personality traits, and perceptions (Rust and Golombok, 2014). In social science re- search, psychometric dimensions are latent con- structs that are known to be important antecedents, moderators, mediators, and consequents for im- portant humanistic behaviors and outcomes. For example, constructs such as threat severity and re- ∗Authors listed alphabetically. sponse efficacy of protective mechanisms are criti- cal psychometric measures of one’s likelihood to avoid security threats (Zahedi et al., 2015). In be- havioral health, psychometric dimensions such as health numeracy, subjective health literacy, trust in physicians, and anxiety visiting the doctor’s office are known to effect various health and wellness outcomes such as future physician visits and all- around well-being (Netemeyer et al., 2020). In electronic commerce, satisfaction with a website’s functional, information, and visual design are cor- related with purchase propensity and customer loy- alty (Cyr, 2008). Similarly, many individualized financial behaviors can be partially explained by fi- nancial literacy and psychological traits (Fernandes et al., 2014). Given the importance of psychometric dimen- sions for understanding behaviors and outcomes in various domains, rigorous data collection protocols and best practices have been developed over the years (Netemeyer et al., 2003). The primary modes of collection involve surveys and interviews. While these techniques afford many benefits …

Content truncated to 500 words