Multi-Task Learning for Sequence Tagging: An Empirical Study

Proceedings of the 27th International Conference on Computational Linguistics, pages 2965–2977 Santa Fe, New Mexico, USA, August 20-26, 2018. 2965 Multi-Task Learning for Sequence Tagging: An Empirical Study Soravit Changpinyo, Hexiang Hu, and Fei Sha Department of Computer Science University of Southern California Los Angeles, CA 90089 schangpi,hexiangh,feisha@usc.edu Abstract We study three general multi-task learning (MTL) approaches on 11 sequence tagging tasks. Our extensive empirical results show that in about 50% of the cases, jointly learning all 11 tasks improves upon either independent or pairwise learning of the tasks. We also show that pairwise MTL can inform us what tasks can benefit others or what tasks can be benefited if they are learned jointly. In particular, we identify tasks that can always benefit others as well as tasks that can always be harmed by others. Interestingly, one of our MTL approaches yields embeddings of the tasks that reveal the natural clustering of semantic and syntactic tasks. Our inquiries have opened the doors to further utilization of MTL in NLP. 1 Introduction Multi-task learning (MTL) has long been studied in the machine learning literature, cf. (Caruana, 1997). The technique has also been popular in NLP, for example, in (Collobert and Weston, 2008; Collobert et al., 2011; Luong et al., 2016). The main thesis underpinning MTL is that solving many tasks together provides a shared inductive bias that leads to more robust and generalizable systems. This is especially appealing for NLP as data for many tasks are scarce — shared learning thus reduces the amount of training data needed. MTL has been validated in recent work, mostly where auxiliary tasks are used to improve the performance on a target task, for example, in sequence tagging (Søgaard and Goldberg, 2016; Bjerva et al., 2016; Plank et al., 2016; Alonso and Plank, 2017; Bingel and Søgaard, 2017). Despite those successful applications, several key issues about the effectiveness of MTL remain open. Firstly, with only a few exceptions, much existing work focuses on “pairwise” MTL where there is a target task and one or several (carefully) selected auxiliary tasks. However, can jointly learning many tasks benefit all of them together? A positive answer will significantly raise the utility of MTL. Secondly, how are tasks related such that one could benefit another? For instance, one plausible intuition is that syntactic and semantic tasks might benefit among their two separate groups though cross-group assis- tance is weak or unlikely. However, such notions have not been put to test thoroughly on a significant number of tasks. In this paper, we address such questions. We investigate learning jointly multiple sequence tagging tasks. Besides using independent single-task learning as a baseline and a popular shared-encoder MTL framework for sequence tagging (Collobert et al., 2011), we propose two variants of MTL, where both the encoder and the decoder could be shared by all tasks. We conduct extensive empirical studies on 11 sequence tagging tasks — we defer the discussion on why we select such tasks to a later section. We demonstrate …

Content truncated to 500 words