GradTS: A Gradient-Based Automatic Auxiliary Task Selection Method Based on Transformer Networks

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5621–5632 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 5621 GradTS: A Gradient-Based Automatic Auxiliary Task Selection Method Based on Transformer Networks Weicheng Ma1, Renze Lou2†, Kai Zhang3†, Lili Wang1, and Soroush Vosoughi4 1,4Department of Computer Science, Dartmouth College 2Department of Computer Science, Zhejiang University City College 3Department of Computer Science and Technology, Tsinghua University 1{first.last}.gr@dartmouth.edu 2marionojump0722@gmail.com 3drogozhang@gmail.com 4soroush.vosoughi@dartmouth.edu Abstract A key problem in multi-task learning (MTL) research is how to select high-quality auxil- iary tasks automatically. This paper presents GradTS, an automatic auxiliary task selec- tion method based on gradient calculation in Transformer-based models. Compared to AU- TOSEM, a strong baseline method, GradTS improves the performance of MT-DNN with a bert-base-cased backend model, from 0.33% to 17.93% on 8 natural language understand- ing (NLU) tasks in the GLUE benchmarks. GradTS is also time-saving since (1) its gradi- ent calculations are based on single-task exper- iments and (2) the gradients are re-used with- out additional experiments when the candidate task set changes. On the 8 GLUE classification tasks, for example, GradTS costs on average 21.32% less time than AUTOSEM with com- parable GPU consumption. Further, we show the robustness of GradTS across various task settings and model selections, e.g. mixed ob- jectives among candidate tasks. The efficiency and efficacy of GradTS in these case studies illustrate its general applicability in MTL re- search without requiring manual task filtering or costly parameter tuning. 1 Introduction MTL (Caruana, 1997) is widely used in NLU re- search to improve the performance of machine learning (ML) models by enlarging the training data size with datapoints related to the primary tasks. However, its efficacy is largely affected by the selection of auxiliary tasks. The auxiliary task selection problem is addressed mainly under two settings. The first setting treats each task as a whole. For example, Bingel and Søgaard (2017) assess task relatedness by exhaustive experiments in all †Work done when interning at the Minds, Machines, and Society Lab at Dartmouth College. task pairs. Nonetheless, high pairwise task corre- lations are often not decisive features for choos- ing auxiliary tasks. Glover and Hokamp (2019) train a policy for task selection through counter- factual estimation, but their learned policy brings improvements only to one out of nine tasks on GLUE benchmarks (Wang et al., 2019). The sec- ond setting subsamples training instances from aux- iliary tasks, e.g. with Bayesian optimization (Ruder and Plank, 2017), but these methods are time- and resource-consuming due to their reliance on multi- task experiments involving all the candidate tasks. AUTOSEM (Guo et al., 2019) combines the two settings into one method, selecting candidate tasks with Thompson sampling and deciding the ratio with which to draw training instances from the se- lected tasks via a Gaussian Process. Despite the higher quality of the auxiliary task sets it gener- ates, AUTOSEM is still costly, similar to Ruder and Plank (2017). To design a better-performing and less costly auxiliary task selection method, …

Content truncated to 500 words