A Joint Many-Task Model: Growing a Neural Network for Multiple NLP Tasks

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1923–1933 Copenhagen, Denmark, September 7–11, 2017. c⃝2017 Association for Computational Linguistics A Joint Many-Task Model: Growing a Neural Network for Multiple NLP Tasks Kazuma Hashimoto∗, Caiming Xiong†, Yoshimasa Tsuruoka, and Richard Socher The University of Tokyo {hassy, tsuruoka}@logos.t.u-tokyo.ac.jp Salesforce Research {cxiong, rsocher}@salesforce.com Abstract Transfer and multi-task learning have traditionally focused on either a single source-target pair or very few, similar tasks. Ideally, the linguistic levels of mor- phology, syntax and semantics would ben- efit each other by being trained in a sin- gle model. We introduce a joint many-task model together with a strategy for succes- sively growing its depth to solve increas- ingly complex tasks. Higher layers in- clude shortcut connections to lower-level task predictions to reflect linguistic hierar- chies. We use a simple regularization term to allow for optimizing all model weights to improve one task’s loss without exhibit- ing catastrophic interference of the other tasks. Our single end-to-end model ob- tains state-of-the-art or competitive results on five different tasks from tagging, pars- ing, relatedness, and entailment tasks. 1 Introduction The potential for leveraging multiple levels of representation has been demonstrated in various ways in the field of Natural Language Processing (NLP). For example, Part-Of-Speech (POS) tags are used for syntactic parsers. The parsers are used to improve higher-level tasks, such as natural lan- guage inference (Chen et al., 2016) and machine translation (Eriguchi et al., 2016). These systems are often pipelines and not trained end-to-end. Deep NLP models have yet shown benefits from predicting many increasingly complex tasks each at a successively deeper layer. Existing models often ignore linguistic hierarchies by predicting ∗ Work was done while the first author was an intern at Salesforce Research. †Corresponding author. CHUNK POS DEP Relatedness encoder Relatedness Entailment encoder Entailment word representation Sentence1 CHUNK POS DEP Relatedness encoder Entailment encoder word representation Sentence2 semantic level syntactic level word level Figure 1: Overview of the joint many-task model predicting different linguistic outputs at succes- sively deeper layers. different tasks either entirely separately or at the same depth (Collobert et al., 2011). We introduce a Joint Many-Task (JMT) model, outlined in Figure 1, which predicts increasingly complex NLP tasks at successively deeper lay- ers. Unlike traditional pipeline systems, our sin- gle JMT model can be trained end-to-end for POS tagging, chunking, dependency parsing, semantic relatedness, and textual entailment, by consider- ing linguistic hierarchies. We propose an adaptive training and regularization strategy to grow this model in its depth. With the help of this strat- egy we avoid catastrophic interference between the tasks. Our model is motivated by Søgaard and Goldberg (2016) who showed that predicting two different tasks is more accurate when performed in different layers than in the same layer (Collobert et al., 2011). Experimental results show that our single model achieves competitive results for all of the five different tasks, demonstrating that us- 1923 ing linguistic hierarchies is more important than handling different tasks in the same …

Content truncated to 500 words