PADA: Example-based Prompt Learning for on-the-fly Adaptation to Unseen Domains

PADA: Example-based Prompt Learning for on-the-fly Adaptation to Unseen Domains Eyal Ben-David∗ Nadav Oved∗ Roi Reichart Technion - Israel Institute of Technology, Israel {eyalbd12@campus.|nadavo@campus.|roiri@}technion.ac.il Abstract Natural Language Processing algorithms have made incredible progress, but they still struggle when applied to out-of-distribution exam- ples. We address a challenging and under- explored version of this domain adaptation problem, where an algorithm is trained on several source domains, and then applied to examples from unseen domains that are un- known at training time. Particularly, no ex- amples, labeled or unlabeled, or any other knowledge about the target domain are avail- able to the algorithm at training time. We present PADA: An example-based autoregres- sive Prompt learning algorithm for on-the-fly Any-Domain Adaptation, based on the T5 lan- guage model. Given a test example, PADA first generates a unique prompt for it and then, conditioned on this prompt, labels the exam- ple with respect to the NLP prediction task. PADA is trained to generate a prompt that is a token sequence of unrestricted length, con- sisting of Domain Related Features (DRFs) that characterize each of the source domains. Intuitively, the generated prompt is a unique signature that maps the test example to a se- mantic space spanned by the source domains. In experiments with 3 tasks (text classifica- tion and sequence tagging), for a total of 14 multi-source adaptation scenarios, PADA sub- stantially outperforms strong baselines.1 1 Introduction Natural Language Processing (NLP) algorithms are gradually achieving remarkable milestones (Devlin et al., 2019; Lewis et al., 2020; Brown et al., 2020). However, such algorithms often rely on the seminal assumption that the training set and the test set come from the same underly- ing distribution. Unfortunately, this assumption ∗Both authors equally contributed to this work. 1Our code and data are available at https://github .com/eyalbd2/PADA. often does not hold since text may emanate from many different sources, each with unique distributional properties. As generalization be- yond the training distribution is still a fundamental challenge, NLP algorithms suffer a significant degradation when applied to out-of-distribution examples. thisDomain Adaptation (DA) explicitly addresses challenge, striving to improve out-of- distribution generalization of NLP algorithms. DA algorithms are trained on annotated data from source domains, to be effectively applied in a variety of target domains. Over the years, con- siderable efforts have been devoted to the DA challenge, focusing on various scenarios where the target domain is known at training time (e.g., through labeled or unlabeled data) but is yet under- represented (Roark and Bacchiani, 2003; Daum´e III and Marcu, 2006; Reichart and Rappoport, 2007; McClosky et al., 2010; Rush et al., 2012; Schnabel and Sch¨utze, 2014). Still, the challenge of adaptation to any possible target domain, which is unknown at training time, is underexplored in DA literature.2 In this work, we focus on adaptation to any target domain, which we consider a ‘‘Holy Grail’’ of DA (§3). Apart from the pronounced intellec- tual challenge, it also presents unique modeling advantages as target-aware algorithms typically require training a separate model for each …

Content truncated to 500 words