Exploring Logically Dependent Multi-task Learning with Causal Inference

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 2213–2225, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 2213 Exploring Logically Dependent Multi-task Learning with Causal Inference Wenqing Chen, Jidong Tian, Liqiang Xiao, Hao He∗, Yaohui Jin∗ MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University State Key Lab of Advanced Optical Communication System and Network, Shanghai Jiao Tong University {wenqingchen, hehao, jinyh}@sjtu.edu.cn Abstract Previous studies have shown that hierarchical multi-task learning (MTL) can utilize task de- pendencies by stacking encoders and outper- form democratic MTL. However, stacking en- coders only considers the dependencies of fea- ture representations and ignores the label de- pendencies in logically dependent tasks. Fur- thermore, how to properly utilize the labels re- mains an issue due to the cascading errors be- tween tasks. In this paper, we view logically dependent MTL from the perspective of causal inference and suggest a mediation assumption instead of the confounding assumption in con- ventional MTL models. We propose a model including two key mechanisms: label transfer (LT) for each task to utilize the labels of all its lower-level tasks, and Gumbel sampling (GS) to deal with cascading errors. In the field of causal inference, GS in our model is essen- tially a counterfactual reasoning process, try- ing to estimate the causal effect between tasks and utilize it to improve MTL. We conduct ex- periments on two English datasets and one Chi- nese dataset. Experiment results show that our model achieves state-of-the-art on six out of seven subtasks and improves predictions’ con- sistency. 1 Introduction Multi-task learning (MTL) has received increas- ing interest with the knowledge transfer potential among related tasks (Caruana, 1997; Ruder et al., 2017; Liu et al., 2019). Recently, hierarchical MTL models (Hashimoto et al., 2017; Sanh et al., 2018) were proposed for tasks with dependencies and achieved better performance than democratic ones. In their models, the encoders of different tasks were stacked. And the proposition is that complex tasks at top layers require deep processes * Corresponding authors X Y m Y n T n T m S X Y m Y n T n T m S X Y m Y n T n T m S (a) Democratic X Y m Y n T n T m S X Y m Y n T n T m S X Y m Y n T n T m S (b) Hierarchical X Y m Y n T n T m S X Y m Y n T n T m S X Y m Y n T n T m S (c) Logically De- pendent Figure 1: Three types of MTL schemes for two tasks tm and tn: (a) Democratic MTL shares an encoder S and owns task-specific encoders T i (i = m or n); (b) Hierarchical MTL stacks those task-specific encoders; (c) Logically dependent MTL further considers the la- bel dependencies by re-utilizing low-level tasks’ labels. to capture semantic richer features, and simple tasks at bottom …

Content truncated to 500 words