Representation Learning Using Multi-Task Deep Neural Networks for Semantic Classification and Information Retrieval

Human Language Technologies: The 2015 Annual Conference of the North American Chapter of the ACL, pages 912–921, Denver, Colorado, May 31 – June 5, 2015. c⃝2015 Association for Computational Linguistics Representation Learning Using Multi-Task Deep Neural Networks for Semantic Classification and Information Retrieval Xiaodong Liu†∗, Jianfeng Gao‡, Xiaodong He‡, Li Deng‡, Kevin Duh† and Ye-yi Wang‡ †Nara Institute of Science and Technology, 8916-5 Takayama, Ikoma, Nara 630-0192, Japan ‡Microsoft Research, One Microsoft Way, Redmond, WA 98052, USA xiaodong-l@is.naist.jp, {jfgao,xiaohe,deng}@microsoft.com kevinduh@is.naist.jp, yeyiwang@microsoft.com Abstract Methods of deep neural networks (DNNs) have recently demonstrated superior perfor- mance on a number of natural language pro- cessing tasks. However, in most previous work, the models are learned based on ei- ther unsupervised objectives, which does not directly optimize the desired task, or single- task supervised objectives, which often suf- fer from insufficient training data. We de- velop a multi-task DNN for learning represen- tations across multiple tasks, not only leverag- ing large amounts of cross-task data, but also benefiting from a regularization effect that leads to more general representations to help tasks in new domains. Our multi-task DNN approach combines tasks of multiple-domain classification (for query classification) and in- formation retrieval (ranking for web search), and demonstrates significant gains over strong baselines in a comprehensive set of domain adaptation. 1 Introduction Recent advances in deep neural networks (DNNs) have demonstrated the importance of learning vector-space representations of text, e.g., words and sentences, for a number of natural language process- ing tasks. For example, the study reported in (Col- lobert et al., 2011) demonstrated significant accu- racy gains in tagging, named entity recognition, and semantic role labeling when using vector space word ∗This research was conducted during the author’s internship at Microsoft Research. representations learned from large corpora. Fur- ther, since these representations are usually in a low- dimensional vector space, they result in more com- pact models than those built from surface-form fea- tures. A recent successful example is the parser by (Chen and Manning, 2014), which is not only accu- rate but also fast. However, existing vector-space representation learning methods are far from optimal. Most pre- vious methods are based on unsupervised objectives such as word prediction for training (Mikolov et al., 2013c; Pennington et al., 2014). Other methods use supervised training objectives on a single task, e.g. (Socher et al., 2013), and thus are often constrained by limited amounts of training data. Motivated by the success of multi-task learning (Caruana, 1997), we propose in this paper a multi-task DNN approach for representation learning that leverages supervised data from many tasks. In addition to the benefit of having more data for training, the use of multi-task also profits from a regularization effect, i.e., reduc- ing overfitting to a specific task, thus making the learned representations universal across tasks. Our contributions are of two-folds: First, we pro- pose a multi-task deep neural network for represen- tation learning, in particular focusing on semantic classification (query classification) and semantic in- formation retrieval (ranking for web search) tasks. …

Content truncated to 500 words