Can Demographic Factors Improve Text Classification? Revisiting Demographic Adaptation in the Age of Transformers

Findings of the Association for Computational Linguistics: EACL 2023, pages 1565–1580 May 2-6, 2023 ©2023 Association for Computational Linguistics Can Demographic Factors Improve Text Classification? Revisiting Demographic Adaptation in the Age of Transformers Chia-Chien Hung1,5, Anne Lauscher2, Dirk Hovy3, Simone Paolo Ponzetto1 and Goran Glavaš4 1Data and Web Science Group, University of Mannheim, Germany 2Data Science Group, University of Hamburg, Germany 3MilaNLP, Bocconi University, Italy 4CAIDAS, University of Würzburg, Germany 5NEC Laboratories Europe GmbH, Heidelberg, Germany {chia-chien.hung, ponzetto}@uni-mannheim.de anne.lauscher@uni-hamburg.de, dirk.hovy@unibocconi.it goran.glavas@uni-wuerzburg.de Abstract Demographic factors (e.g., gender or age) shape our language. Previous work showed that incorporating demographic factors can consistently improve performance for various NLP tasks with traditional NLP models. In this work, we investigate whether these pre- vious findings still hold with state-of-the-art pretrained Transformer-based language mod- els (PLMs). We use three common specializa- tion methods proven effective for incorporat- ing external knowledge into pretrained Trans- formers (e.g., domain-specific or geographic knowledge). We adapt the language representa- tions for the demographic dimensions of gender and age, using continuous language modeling and dynamic multi-task learning for adaptation, where we couple language modeling objectives with the prediction of demographic classes. Our results, when employing a multilingual PLM, show substantial gains in task perfor- mance across four languages (English, German, French, and Danish), which is consistent with the results of previous work. However, con- trolling for confounding factors – primarily do- main and language proficiency of Transformer- based PLMs – shows that downstream perfor- mance gains from our demographic adaptation do not actually stem from demographic knowl- edge. Our results indicate that demographic specialization of PLMs, while holding promise for positive societal impact, still represents an unsolved problem for (modern) NLP. 1 Introduction Demographic factors like social class, education, income, age, or gender, categorize people into spe- cific groups or populations. At the same time, demographic factors both shape and are reflected in our language (e.g., Trudgill, 2000; Eckert and McConnell-Ginet, 2013). A large body of work focused on modeling demographic language vari- ation, especially the correlations between words and demographic factors (Bamman et al., 2014; Garimella et al., 2017; Welch et al., 2020, inter alia). In a similar vein, Volkova et al. (2013) and Hovy (2015) demonstrated that explicitly incorpo- rating demographic information in language repre- sentations improves performance on downstream NLP tasks, e.g., topic classification or sentiment analysis. However, these observations rely on ap- proaches that leverage gender-specific lexica to spe- cialize word embeddings and text encoders (e.g., re- current networks) that have not been pretrained for (general purpose) language understanding. To date, the benefits of demographic specialization have not been tested with Transformer-based (Vaswani et al., 2017) pretrained language models (PLMs), which have been shown to excel on the vast majority of NLP tasks and even surpass human performance in some cases (Wang et al., 2018). More recent studies focus mainly on monolin- gual English datasets and introduce demographic features in task-specific fine-tuning (Voigt et al., 2018; Buechel et al., 2018), which limits the bene- fits of demographic knowledge …

Content truncated to 500 words