Multi-Task Deep Neural Networks for Natural Language Understanding

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4487–4496 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 4487 Multi-Task Deep Neural Networks for Natural Language Understanding Xiaodong Liu∗1, Pengcheng He∗2, Weizhu Chen2, Jianfeng Gao1 1 Microsoft Research 2 Microsoft Dynamics 365 AI {xiaodl,penhe,wzchen,jfgao}@microsoft.com Abstract In this paper, we present a Multi-Task Deep Neural Network (MT-DNN) for learning rep- resentations across multiple natural language understanding (NLU) tasks. MT-DNN not only leverages large amounts of cross-task data, but also benefits from a regularization ef- fect that leads to more general representations to help adapt to new tasks and domains. MT- DNN extends the model proposed in Liu et al. (2015) by incorporating a pre-trained bidirec- tional transformer language model, known as BERT (Devlin et al., 2018). MT-DNN ob- tains new state-of-the-art results on ten NLU tasks, including SNLI, SciTail, and eight out of nine GLUE tasks, pushing the GLUE bench- mark to 82.7% (2.2% absolute improvement) 1. We also demonstrate using the SNLI and Sc- iTail datasets that the representations learned by MT-DNN allow domain adaptation with substantially fewer in-domain labels than the pre-trained BERT representations. The code and pre-trained models are publicly available at https://github.com/namisan/mt-dnn. 1 Introduction Learning vector-space representations of text, e.g., words and sentences, is fundamental to many nat- ural language understanding (NLU) tasks. Two popular approaches are multi-task learning and language model pre-training. In this paper we combine the strengths of both approaches by proposing a new Multi-Task Deep Neural Network (MT-DNN). Multi-Task Learning (MTL) is inspired by hu- man learning activities where people often apply the knowledge learned from previous tasks to help learn a new task (Caruana, 1997; Zhang and Yang, 2017). For example, it is easier for a person who knows how to ski to learn skating than the one who ∗Equal Contribution. 1As of February 25, 2019 on the latest GLUE test set. does not. Similarly, it is useful for multiple (re- lated) tasks to be learned jointly so that the knowl- edge learned in one task can benefit other tasks. Recently, there is a growing interest in applying MTL to representation learning using deep neu- ral networks (DNNs) (Collobert et al., 2011; Liu et al., 2015; Luong et al., 2015; Xu et al., 2018; Guo et al., 2018; Ruder12 et al., 2019) for two reasons. First, supervised learning of DNNs re- quires large amounts of task-specific labeled data, which is not always available. MTL provides an effective way of leveraging supervised data from many related tasks. Second, the use of multi-task learning profits from a regularization effect via al- leviating overfitting to a specific task, thus making the learned representations universal across tasks. In contrast to MTL, language model pre- training has shown to be effective for learning universal language representations by leveraging large amounts of unlabeled data. A recent sur- vey is included in Gao et al. (2018). Some of the most prominent examples are ELMo (Peters et al., 2018), GPT (Radford …

Content truncated to 500 words