Divergence-Based Domain Transferability for Zero-Shot Classification

Findings of the Association for Computational Linguistics: EACL 2023, pages 1649–1654 May 2-6, 2023 ©2023 Association for Computational Linguistics Divergence-Based Domain Transferability for Zero-Shot Classification Alexander Pugantsov and Richard McCreadie School of Computing Science, University of Glasgow, UK Abstract Transferring learned patterns from pretrained neural language models has been shown to sig- nificantly improve effectiveness across a variety of language-based tasks, meanwhile further tuning on intermediate tasks has been demon- strated to provide additional performance benefits, provided the intermediate task is suffi- ciently related to the target task. However, how to identify related tasks is an open problem, and brute-force searching effective task combi- nations is prohibitively expensive. Hence, the question arises, are we able to improve the effec- tiveness and efficiency of tasks with no training examples through selective fine-tuning? In this paper, we explore statistical measures that approximate the divergence between domain representations as a means to estimate whether tuning using one task pair will exhibit perfor- mance benefits over tuning another. This esti- mation can then be used to reduce the number of task pairs that need to be tested by eliminat- ing pairs that are unlikely to provide benefits. Through experimentation over 58 tasks and over 6,600 task pair combinations, we demon- strate that statistical measures can distinguish effective task pairs, and the resulting estimates can reduce end-to-end runtime by up to 40%. 1 Introduction As the accuracy of neural models continues to increase, so does the computational cost of training and storing them. One approach of mitigating such cost is through using pretrained models to enhance performance on a downstream task, a paradigm commonly referred to as transfer learning. How- ever, when and why transfer learning works is not concretely understood. Traditionally, selecting the best settings, i.e. tasks and hyperparameters, for transfer often involves an extensive trial-and-error process over many combinations and can quickly make the prospect of applying transfer learning undesirable. As such, it would be valuable to estimate whether a task pair combination will be effective pre-training, i.e. estimate the transferability of a source task to a target task. The most optimal transferability metric would be resource-efficient, such that it is capable of ac- curately predicting the final performance of the model whilst minimising the amount of process- ing required to compute it. To this end, several works (Van Asch and Daelemans, 2010; Ruder and Plank, 2017; Ramesh Kashyap et al., 2021) have focused on estimating transferability prior to fine- tuning, using statistical measures of divergence between the underlying feature spaces of model pairs. Domain divergence measures are used to produce a notion of distance between pairs of do- mains by comparing their representations and have seen significant usage in works which investigate the correlation between their estimations and per- formance change (Van Asch and Daelemans, 2010; Ramesh Kashyap et al., 2021). Subsequent transfer learning works have also demonstrated that competitive model performance can be achieved on some target tasks even if no training samples for that task are avail- able, an approach …

Content truncated to 500 words