Empirical Evaluation of Pre-trained Transformers for Human-Level NLP: The Role of Sample Size and Dimensionality

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4515–4532 June 6–11, 2021. ©2021 Association for Computational Linguistics 4515 Empirical Evaluation of Pre-trained Transformers for Human-Level NLP: The Role of Sample Size and Dimensionality Adithya V Ganesan, Matthew Matero, Aravind Reddy Ravula, Huy Vu, and H. Andrew Schwartz Department of Computer Science, Stony Brook University {avirinchipur, mmatero, aravula, hvu, has}@cs.stonybrook.edu Abstract In human-level NLP tasks, such as predicting mental health, personality, or demographics, the number of observations is often smaller than the standard 768+ hidden state sizes of each layer within modern transformer-based language models, limiting the ability to effec- tively leverage transformers. Here, we provide a systematic study on the role of dimension re- duction methods (principal components anal- ysis, factorization techniques, or multi-layer auto-encoders) as well as the dimensionality of embedding vectors and sample sizes as a function of predictive performance. We first find that fine-tuning large models with a lim- ited amount of data pose a significant difficulty which can be overcome with a pre-trained di- mension reduction regime. RoBERTa con- sistently achieves top performance in human- level tasks, with PCA giving benefit over other reduction methods in better handling users that write longer texts. Finally, we observe that a majority of the tasks achieve results compara- ble to the best performance with just 1 12 of the embedding dimensions. 1 Introduction Transformer based language models (LMs) have quickly become the foundation for accurately ap- proaching many tasks in natural language process- ing (Vaswani et al., 2017; Devlin et al., 2019). Ow- ing to their success is their ability to capture both syntactic and semantic information (Tenney et al., 2019), modeled over large, deep attention-based networks (transformers) with hidden state sizes on the order of 1000 over 10s of layers (Liu et al., 2019; Gururangan et al., 2020). In total such mod- els typically have from hundreds of millions (De- vlin et al., 2019) to a few billion parameters (Raffel et al., 2020). However, the size of such models presents a challenge for tasks involving small num- bers of observations, such as for the growing num- ber of tasks focused on human-level NLP. Human-level NLP tasks, rooted in computational social science, focus on making predictions about people from their language use patterns. Some of the more common tasks include age and gender prediction (Sap et al., 2014; Morgan-Lopez et al., 2017) , personality (Park et al., 2015; Lynn et al., 2020), and mental health prediction (Coppersmith et al., 2014; Guntuku et al., 2017; Lynn et al., 2018). Such tasks present an interesting challenge for the NLP community to model the people behind the language rather than the language itself, and the so- cial scientific community has begun to see success of such approaches as an alternative or supplement to standard psychological assessment techniques like questionnaires (Kern et al., 2016; Eichstaedt et al., 2018). Generally, such work is helping to embed NLP in a greater social and …

Content truncated to 500 words