Learning What to Share: Leaky Multi-Task Network for Text Classification

Proceedings of the 27th International Conference on Computational Linguistics, pages 2055–2065 Santa Fe, New Mexico, USA, August 20-26, 2018. 2055 Learning What to Share: Leaky Multi-Task Network for Text Classification Liqiang Xiao1,2, Honglun Zhang1,2, Wenqing Chen1,2, Yongkun Wang3, Yaohui Jin1,2 1 State Key Lab of Advanced Optical Communication System and Network, Shanghai Jiao Tong University 2 Artificial Intelligence Institute, Shanghai Jiao Tong University 3 Network and Information Center, Shanghai Jiao Tong University {jinyh}@sjtu.edu.cn Abstract Neural network based multi-task learning has achieved great success on many NLP problems, which focuses on sharing knowledge among tasks by linking some layers to enhance the perfor- mance. However, most existing approaches suffer from the interference between tasks because they lack of selection mechanism for feature sharing. In this way, the feature spaces of tasks may be easily contaminated by useless features borrowed from others, which will confuse the mod- els for making correct prediction. In this paper, we propose a multi-task convolutional neural network with the Leaky Unit, which has memory and forgetting mechanism to filter the feature flows between tasks. Experiments on five different datasets for text classification validate the benefits of our approach. 1 Introduction Convolutional neural network (CNN) models have achieved impressive results on many natural language processing (NLP) tasks, which use convolving filters to extract local features and have been successfully applied in computer vision field. In recent years, increasing works transfer CNNs to natural language processing tasks, such as representation learning (Liu et al., 2015), information retrieval (Shen et al., 2014), text classification (Kalchbrenner et al., 2014), etc. In particular, feed-forward CNNs with word embedding have been proven to be a relatively simple yet powerful kind of models for text classification (Kim, 2014). However, the reliance on large-scale corpus has been a formidable constraint for deep neural networks (DNNs) based methods due to their numerous parameters. It costs a lot for a large-scale dataset to train the huge volume of parameters, because constructing a large-scale labeled dataset is extremely labor- intensive. To solve this problem, these models usually employ a pre-trained phase to map words into vectors with semantic implication (Collobert et al., 2011), which just introduces extra knowledge and does not directly optimize the target task. The problem of insufficiency for annotated resources is not intrinsically solved either. Multi-task learning (MTL) can implicitly increase the corpus size and create a synergy effect among datasets (Caruana, 1997). By learning related tasks in parallel, MTL has the ability to exploit the relations between similar tasks to benefit each other, and eventually improves the performance for classification. In addition, models handling multiple tasks also benefit from a regularization effect to decrease the overfitting. It can help the models to learn a more universal representation for text sequences. Inspired by this, more DNN-based models (Collobert and Weston, 2008; Liu et al., 2015; Liu et al., 2016) utilize multi-task learning to improve their performance. Traditionally, multi-task learning only shares a few shallow layers among tasks, so only low-level knowledge is exchanged to benefit …

Content truncated to 500 words