Tchebycheff Procedure for Multi-task Text Classification

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4217–4226 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 4217 Tchebycheff Procedure for Multi-task Text Classification Yuren Mao†, Shuang Yun‡, Weiwei Liu‡∗, Bo Du‡ ‡School of Computer Science, Wuhan University †School of Computer Science and Engineering, University of New South Wales yuren.mao@unsw.edu.au,liuweiwei863@gmail.com, {yunshuang2014, dubo}@whu.edu.cn Abstract Multi-task Learning methods have achieved significant progress in text classification. How- ever, existing methods assume that multi-task text classification problems are convex multi- objective optimization problems, which is un- realistic in real-world applications. To ad- dress this issue, this paper presents a novel Tchebycheff procedure to optimize the multi- task classification problems without any con- vex assumption. The extensive experiments back up our theoretical analysis and validate the superiority of our proposals. 1 Introduction Multi-task Learning (MTL) aims to learn multi- ple related tasks simultaneously, and obtain better performance than learning each task independently by setting inductive bias across tasks. (Caruana, 1993; Bakker and Heskes, 2003; Ben-David and Schuller, 2003; Ando and Zhang, 2005). It has achieved great success in various applications rang- ing from computer vision (Kendall et al., 2018) to text classification (Liu et al., 2016, 2017; Xiao et al., 2018). Existing MTL methods for text classification, usually set up the inductive bias across tasks by designing a parameterized hypothesis class that shares some parameters across tasks (e.g. shares some hidden layers in a Neural Network), and cast the multi-task text classification problem as a multi- objective optimization problem. L1-metric method is one of the most popular strategies for solving the multi-objective optimization problem. Specifically, it learns the parameters by minimizing a weighted linear combination of per-task losses. And this method is able to find an arbitrary Pareto optimal solution in the Pareto set if the problem is con- vex. Unfortunately, for a non-convex problem, this *Corresponding author. Empirical Risk of Task 1 Empirical Risk of Task 2 Pareto Front (a) Convex case. Empirical Risk of Task 1 Empirical Risk of Task 2 Pareto Front (b) Non-convex case. Figure 1: Graphical interpretation of Pareto optimiza- tion for weighted linear combination based MTL. The points of tangency between the line of linear combina- tion and Pareto front are Pareto optimal points. (a). in the convex case, all the Pareto optimal points are achievable; (b). in the non-convex case, the Pareto op- timal points located at the concave part of the Pareto front are unachievable. method excludes many Pareto optimal solutions from its search scope. To illustrate the issue, it is instructive to consider a 2-tasks learning case shown as Figure 1. From Figure 1, we can see that for a non-convex problem, the Pareto points located at the concave part of the Pareto front are unachievable. According to the uniform conver- gence properties of MTL (Baxter, 2000), the exclu- sion of Pareto optimal solutions may degenerate the generalization performance of multi-task text classification. To address the non-convexity problems, this pa- per proposes a novel Tchebycheff procedure to improve the …

Content truncated to 500 words