Mixed-effects transformers for hierarchical adaptation

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3944 - 3954 December 7-11, 2022 ©2022 Association for Computational Linguistics Mixed-Effects Transformers for Hierarchical Adaptation Julia White Electrical Engineering Stanford University jiwhite@stanford.edu Noah D. Goodman Computer Science, Psychology Stanford University ndg@stanford.edu Robert D. Hawkins Psychology Princeton University rdhawkins@princeton.ed Abstract Language differs dramatically from context to context. To some degree, large language mod- els like GPT-3 account for such variation by conditioning on strings of initial input text, or prompts. However, prompting can be ineffec- tive when contexts are sparse, out-of-sample, or extra-textual. In this paper, we introduce the mixed-effects transformer (MET), a novel ap- proach for learning hierarchically-structured prefixes— lightweight modules prepended to an input sequence— to account for struc- tured variation in language use. Specifically, we show how the popular class of mixed- effects regression models may be extended to transformer-based architectures using a regu- larized prefix-tuning procedure with dropout. We evaluate this approach on several domain- adaptation benchmarks, finding that it learns contextual variation from minimal data while generalizing well to unseen contexts. 1 Introduction While certain aspects of language use are nearly universal – such as basic grammatical acceptability (Warstadt et al., 2019; Linzen and Baroni, 2021) or simple lexical judgements (Wang et al., 2019) – these often seem to be the exception that proves the rule. Contextual variation is ubiquitous in lan- guage, where predictions may differ as a function of speaker identity (Blodgett et al., 2016; Yang and Eisenstein, 2017; Ostapenko et al., 2022), loca- tion (Hofmann et al., 2022), time (Lazaridou et al., 2021; Sawhney et al., 2020; Schlechtweg et al., 2019; Röttger and Pierrehumbert, 2021), or usage domain (Dai et al., 2020; Nguyen et al., 2020; Lee et al., 2020). Although such variation has long been recognized in psycholinguistics (Clark, 1998) and sociolinguistics (Nardy et al., 2013; Eckert, 2012), the dominant approach in modern NLP has been to train monolithic models (Flek, 2020; Hovy, 2015) and fine-tune for individual domains if necessary (e.g. Daume III and Marcu, 2006). Figure 1: In the mixed-effects transformer (MET), pa- rameters of a pretrained transformer are frozen (solid border) while prefixes are adapted to different contex- tual features (dashed border). Recent large language models (LLMs) like GPT-3 (Brown et al., 2020; Bommasani et al., 2021) have begun to provide a more systematic approach for handling context-specific variance. By adding relevant contextual information to the text input (i.e. prompting), these models have been able to account for known demographic informa- tion such as the speaker’s age, gender, or country of origin (Ostapenko et al., 2022). However, it is less clear how to use prompting when context is extra-textual, contains multiple features, or lies out- side the training distribution. For example, LLMs trained prior to the COVID-19 pandemic failed catastrophically on the torrent of new tweets and medical papers (Feldman et al., 2021; Zeng et al., 2020; Luu et al., 2021). In these cases, some degree of online adaptation is required. One particularly promising adapta- …

Content truncated to 500 words