How Transferable are Neural Networks in NLP Applications?

Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 479–489, Austin, Texas, November 1-5, 2016. c⃝2016 Association for Computational Linguistics How Transferable are Neural Networks in NLP Applications? Lili Mou,1 Zhao Meng,1 Rui Yan,2 Ge Li,1,† Yan Xu,1,∗ Lu Zhang,1 Zhi Jin1,† 1Key Laboratory of High Confidence Software Technologies (Peking University), MoE, China Institute of Software, Peking University, China †Corresponding authors 2Insitute of Computer Science and Technology of Peking University, China {doublepower.mou,rui.yan.peking}@gmail.com,zhaomeng.pku@outlook.com {lige,xuyan14,zhanglu,zhijin}@sei.pku.edu.cn Abstract Transfer learning is aimed to make use of valuable knowledge in a source domain to help model performance in a target domain. It is particularly important to neural networks, which are very likely to be overfitting. In some fields like image processing, many stud- ies have shown the effectiveness of neural network-based transfer learning. For neural NLP, however, existing studies have only ca- sually applied transfer learning, and conclu- sions are inconsistent. In this paper, we con- duct systematic case studies and provide an illuminating picture on the transferability of neural networks in NLP.1 1 Introduction Transfer learning, or sometimes known as domain adaptation,2 plays an important role in various nat- ural language processing (NLP) applications, espe- cially when we do not have large enough datasets for the task of interest (called the target task T ). In such scenarios, we would like to transfer or adapt knowledge from other domains (called the source domains/tasks S) so as to mitigate the problem of overfitting and to improve model performance in T . For traditional feature-rich or kernel-based mod- els, researchers have developed a variety of ele- gant methods for domain adaptation; examples in- clude EasyAdapt (Daum´e III, 2007; Daum´e III et ∗Yan Xu is currently a research scientist at Inveno Co., Ltd. 1Code released on https://sites.google.com/site/transfernlp/ 2In this paper, we do not distinguish the conceptual differ- ence between transfer learning and domain adaptation. Do- main—in the sense we use throughout this paper—is defined by datasets. al., 2010), instance weighting (Jiang and Zhai, 2007; Foster et al., 2010), and structural correspondence learning (Blitzer et al., 2006; Prettenhofer and Stein, 2010). Recently, deep neural networks are emerging as the prevailing technical solution to almost every field in NLP. Although capable of learning highly nonlinear features, deep neural networks are very prone to overfitting, compared with traditional meth- ods. Transfer learning therefore becomes even more important. Fortunately, neural networks can be trained in a transferable way by their incremental learning nature: we can directly use trained (tuned) parameters from a source task to initialize the net- work in the target task; alternatively, we may also train two tasks simultaneously with some parame- ters shared. But their performance should be verified by empirical experiments. Existing studies have already shown some evi- dence of the transferability of neural features. For example, in image processing, low-level neural lay- ers closely resemble Gabor filters or color blobs (Zeiler and Fergus, 2014; Krizhevsky et al., 2012); they can be transferred well to different tasks. Don- ahue et al. (2014) suggest …

Content truncated to 500 words