Multi-Task Learning of Pairwise Sequence Classification Tasks over Disparate Label Spaces

Proceedings of NAACL-HLT 2018, pages 1896–1906 New Orleans, Louisiana, June 1 - 6, 2018. c⃝2018 Association for Computational Linguistics Multi-task Learning of Pairwise Sequence Classification Tasks Over Disparate Label Spaces Isabelle Augenstein1∗, Sebastian Ruder23∗, Anders Søgaard1 1Department of Computer Science, University of Copenhagen, Denmark 2Insight Research Centre, National University of Ireland, Galway 3Aylien Ltd., Dublin, Ireland {augenstein|soegaard}@di.ku.dk, sebastian@ruder.io Abstract We combine multi-task learning and semi- supervised learning by inducing a joint embed- ding space between disparate label spaces and learning transfer functions between label em- beddings, enabling us to jointly leverage un- labelled data and auxiliary, annotated datasets. We evaluate our approach on a variety of se- quence classification tasks with disparate la- bel spaces. We outperform strong single and multi-task baselines and achieve a new state- of-the-art for topic-based sentiment analysis. 1 Introduction Multi-task learning (MTL) and semi-supervised learning are both successful paradigms for learn- ing in scenarios with limited labelled data and have in recent years been applied to almost all ar- eas of NLP. Applications of MTL in NLP, for ex- ample, include partial parsing (Søgaard and Gold- berg, 2016), text normalisation (Bollman et al., 2017), neural machine translation (Luong et al., 2016), and keyphrase boundary classification (Au- genstein and Søgaard, 2017). Contemporary work in MTL for NLP typically focuses on learning representations that are useful across tasks, often through hard parameter shar- ing of hidden layers of neural networks (Collobert et al., 2011; Søgaard and Goldberg, 2016). tasks share optimal hypothesis classes at the levelIf of these representations, MTL leads to improve- ments (Baxter, 2000). However, while sharing hidden layers of neural networks is an effective regulariser (Søgaard and Goldberg, 2016), we po- tentially loose synergies between the classification functions trained to associate these representations with class labels. This paper sets out to build an ar- chitecture in which such synergies are exploited, ⋆The first two authors contributed equally. with an application to pairwise sequence classifi- cation tasks. Doing so, we achieve a new state of the art on topic-based sentiment analysis. For many NLP tasks, disparate label sets are weakly correlated, e.g. part-of-speech tags corre- late with dependencies (Hashimoto et al., 2017), sentiment correlates with emotion (Felbo et al., 2017; Eisner et al., 2016), etc. We thus propose to induce a joint label embedding space (visualised in Figure 2) using a Label Embedding Layer that allows us to model these relationships, which we show helps with learning. In addition, for tasks where labels are closely related, we should be able to not only model their relationship, but also to directly estimate the cor- responding label of the target task based on auxil- iary predictions. To this end, we propose to train a Label Transfer Network (LTN) jointly with the model to produce pseudo-labels across tasks. The LTN can be used to label unlabelled and auxiliary task data by utilising the ‘dark knowl- edge’ (Hinton et al., 2015) contained in auxil- iary model predictions. This pseudo-labelled data is then incorporated into the model via semi- supervised …

Content truncated to 500 words