When is multitask learning effective? Semantic sequence prediction under varying data conditions

Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 44–53, Valencia, Spain, April 3-7, 2017. c⃝2017 Association for Computational Linguistics When is multitask learning effective? Semantic sequence prediction under varying data conditions Héctor Martínez Alonso♠ Barbara Plank♥ ♥ Center for Language and Cognition, University of Groningen, The Netherlands ♠ Univ. Paris Diderot, Sorbonne Paris Cité – Alpage, INRIA, France hector.martinez-alonso@inria.fr,b.plank@rug.nl Abstract Multitask learning has been applied suc- cessfully to a range of tasks, mostly mor- phosyntactic. However, little is known on when MTL works and whether there are data characteristics that help to deter- mine its success. In this paper we evalu- ate a range of semantic sequence labeling tasks in a MTL setup. We examine differ- ent auxiliary tasks, amongst which a novel setup, and correlate their impact to data- dependent conditions. Our results show that MTL is not always effective, signif- icant improvements are obtained only for 1 out of 5 tasks. When successful, auxil- iary tasks with compact and more uniform label distributions are preferable. 1 Introduction The recent success of recurrent neural networks (RNNs) for sequence prediction has raised a great deal of interest, which has lead researchers to propose competing architectures for several language-processing tasks. These architectures of- ten rely on multitask learning (Caruana, 1997). Multitask learning (MTL) has been applied with success to a variety of sequence-prediction tasks including chunking and tagging (Collobert et al., 2011; Søgaard and Goldberg, 2016; Bjerva et al., 2016; Plank, 2016), name error detection (Cheng et al., 2015) and machine translation (Luong et al., 2016). However, little is known about MTL for tasks which are more semantic in nature, i.e., tasks that aim at labeling some aspect of the mean- ing of words (Cruse, 1986), instead their mor- phosyntactic behavior. In fact, results on seman- tic tasks are either mixed (Collobert et al., 2011) or, due to the file drawer bias (Rosenthal, 1979), simply not reported. There is no prior study—to the best of our knowledge—that compares data- dependent conditions with performance measures to shed some light on when MTL works for se- mantic sequence prediction. Besides any varia- tion in annotation and conceptualization, the la- bel distributions of such semantic tasks tends to be very different to the characteristic distributions expected in more frequently studied morphosyn- tactic tasks such as POS-tagging. The main contribution of this work is an eval- uation of MTL on semantic sequence predic- tion on data-dependent conditions. We derive characteristics of datasets that make them favor- able for MTL, by comparing performance with information-theoretical metrics of the label fre- quency distribution. We use an off-the-shelf state-of-the-art archi- tecture based on bidirectional Long-Short Term Memory (LSTM) models (Section 3) and evaluate its behavior on a motivated set of main and auxil- iary tasks. We gauge the performance of the MTL setup (Section 4) in the following ways: i) we experiment with different combinations of main and auxiliary tasks, using semantic tasks as main task …

Content truncated to 500 words