When does deep multi-task learning work for loosely related document classification tasks?

Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 1–8 Brussels, Belgium, November 1, 2018. c⃝2018 Association for Computational Linguistics 1 When does deep multi-task learning work for loosely related document classification tasks? Emma Kerinec ´Ecole Normale Sup´erieure de Lyon Lyon, France emma.kerinec@ens-lyon.fr Anders Søgaard Dpt. of Computer Science University of Copenhagen soegaard@di.ku.dk Chlo´e Braud∗ Universit´e de Lorraine, CNRS, LORIA Nancy, France chloe.braud@loria.fr Abstract This work aims to contribute to our under- standing of when multi-task learning through parameter sharing in deep neural networks leads to improvements over single-task learn- ing. We focus on the setting of learning from loosely related tasks, for which no theoretical guarantees exist. We therefore approach the question empirically, studying which proper- ties of datasets and single-task learning char- acteristics correlate with improvements from multi-task learning. We are the first to study this in a text classification setting and across more than 500 different task pairs. 1 Introduction Multi-task learning is a set of techniques for ex- ploiting synergies between related tasks, and in natural language processing (NLP), where there is an overwhelming number of related problems, and different ways to represent these problems, multi- task learning seems well-motivated. Since multi- task learning, by exploiting related tasks, also re- duces the need for labeled data, multi-task learn- ing is also often seen as a way to obtain more ro- bust NLP for more domains and languages. Multi-task learning has seen a revival in recent years, amplified by the success of deep learning techniques. Multi-task learning algorithms have been proven to lead to better performance for sim- ilar tasks, e.g., Baxter and others (2000), such as models of individual patients in health care, but recently multi-task learning has been applied to more loosely related sets of tasks in artificial in- telligence. Examples include machine translation and syntactic parsing (Kaiser et al., 2017) or fixa- tion prediction and sentence compression (Klerke, Goldberg, and Søgaard, 2016). Reported results ∗This work was done, when the third author was affiliated with Dpt. of Computer Science, University of Copenhagen. have been promising, but in the case of loosely re- lated tasks, often also with different label spaces, we have no guarantees that multi-task learning will work. Recent studies have tried to study empirically when multi-task learning leads to improvements (Alonso and Plank, 2017; Bingel and Søgaard, 2017). These preliminary studies have argued – Bingel and Søgaard (2017) most clearly – that multi-task learning is particularly effective when the target task otherwise plateaus faster than the auxiliary task. This study compliments these stud- ies, considering new tasks and architectures, and our findings are largely supportive of this conclu- sion. In text classification, however, performance also depends crucially on the divergence between the marginal distributions of words in the target and auxiliary task. Document classification comes in many differ- ent flavors, including spam detection, sentiment analysis, customer support ticket routing, and di- agnosis support based on patient records, but in this paper we focus on topic-level multi-way …

Content truncated to 500 words