Improving Gradient Trade-offs between Tasks in Multi-task Text Classification

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 2565–2579 July 9-14, 2023 ©2023 Association for Computational Linguistics Improving Gradient Trade-offs between Tasks in Multi-task Text Classification Heyan Chai1, Jinhao Cui1, Ye Wang2, Min Zhang1, Binxing Fang1,3 and Qing Liao1,3∗ 1 Harbin Institute of Technology, Shenzhen, China 2 National University of Defense Technology, China 3 Peng Cheng Laboratory, Shenzhen, China {chaiheyan,cuijinhao}@stu.hit.edu.cn, ye.wang@nudt.edu.cn zhangmin2021@hit.edu.cn, fangbx@cae.cn, liaoqing@hit.edu.cn Abstract Multi-task learning (MTL) has emerged as a promising approach for sharing inductive bias across multiple tasks to enable more ef- ficient learning in text classification. How- ever, training all tasks simultaneously often yields degraded performance of each task than learning them independently, since different tasks might conflict with each other. Exist- ing MTL methods for alleviating this issue is to leverage heuristics or gradient-based algo- rithm to achieve an arbitrary Pareto optimal trade-off among different tasks. In this paper, we present a novel gradient trade-off approach to mitigate the task conflict problem, dubbed GetMTL, which can achieve a specific trade- off among different tasks nearby the main ob- jective of multi-task text classification (MTC), so as to improve the performance of each task simultaneously. The results of extensive exper- iments on two benchmark datasets back up our theoretical analysis and validate the superior- ity of our proposed GetMTL. 1 Introduction Multi-task Learning (MTL), which aims to learn a single model that can tackle multiple correlated but different tasks simultaneously, makes multi- ple tasks benefit from each other and obtain supe- rior performance over learning each task indepen- dently (Caruana, 1997; Ruder, 2017; Liu et al., 2015; Mao et al., 2020). By discovering shared information/structure across the tasks, it has gained attention in many areas of research and industrial communities, such as computer vision (Misra et al., 2016; Gao et al., 2019; Yogamani et al., 2019; Sun et al., 2020) and text classification (Liu et al., 2017; Xiao et al., 2018; Mao et al., 2021, 2022). However, it is observed in multi-task text clas- sification (MTC) scenarios that some tasks could conflict with each other, which may be reflected via conflicting gradients or dominating gradients (Yu ∗ Corresponding Author Task 1 Error Task 2 Error Pareto Front Pareto Solutions (a) Pareto Optimality Task 1 Error Task 2 Error Pareto Front Our Solutions Average Loss (b) GetMTL (Ours) Figure 1: Graphical interpretation of existing Pareto multi-task learning methods for a two-task learning problem. (a) Pareto optimal solutions are arbitrary and uncontrollable. (b) Our GetMTL can find the specific solutions nearby the main objective (Average loss). et al., 2020; Vandenhende et al., 2022), leading to the degraded performance of MTL due to poor training. How to make a proper trade-off among jointing different tasks in MTC is a difficult prob- lem. Recently, several methods have been proposed to mitigate gradient conflicts issue via both loss balance (linear weighted scalarization) such as ho- moscedastic uncertainty (Kendall et al., 2018) and task variance regularization (Mao et al., 2021), and gradient balance …

Content truncated to 500 words