SC-LSTM: Learning Task-Specific Representations in Multi-Task Learning for Sequence Labeling

Proceedings of NAACL-HLT 2019, pages 2396–2406 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 2396 SC-LSTM: Learning Task-Specific Representations in Multi-Task Learning for Sequence Labeling Peng Lu RALI-DIRO Université de Montréal Montréal, Canada peng.lu@umontreal.ca Ting Bai˚ School of Information Renmin University of China Beijing, China baiting@ruc.edu.cn Philippe Langlais RALI-DIRO Université de Montréal Montréal, Canada felipe@iro.umontreal.ca Abstract Multi-task learning (MTL) has been studied recently for sequence labeling. Typically, aux- iliary tasks are selected specifically in order to improve the performance of a target task. Jointly learning multiple tasks in a way that benefit all of them simultaneously can in- crease the utility of MTL. In order to do so, we propose a new LSTM cell which con- tains both shared parameters that can learn from all tasks, and task-specific parameters that can learn task specific information. We name it a Shared-Cell Long-Short Term Mem- ory (SC-LSTM). Experimental results on three sequence labeling benchmarks (named-entity recognition, text chunking, and part-of-speech tagging) demonstrate the effectiveness of our SC-LSTM cell. 1 Introduction As one of the fundamental tasks in NLP, se- quence labeling has been studied for years. Before the blooming of neural network methods, hand- crafted features were widely used in traditional approaches like CRFs, HMMs, and maximum en- tropy classifiers (Lafferty et al., 2001; McCallum et al., 2000; McCallum and Li, 2003; Florian et al., 2003). However, applying them to different tasks or domains is hard. Recently, instead of using handcrafted features, end-to-end neural network based systems have been developed for sequence labeling tasks, such as LSTM-CNN (Chiu and Nichols, 2015), LSTM-CRF (Huang et al., 2015; Lample et al., 2016), and LSTM-CNN-CRF (Ma and Hovy, 2016). These models utilize LSTM to encode the global information of a sentence into a word-level representation of its tokens, which avoids manual feature engineering. Moreover, by incorporating a character-level representation of tokens, these models further improve. ˚ Co-first author. In many such studies, though, neural network models are trained toward a single task in a super- vised way by making use of relatively small anno- tated training material. Jointly learning multiple tasks can reduce the risk of over-fitting to one task, and many attempts have been made at doing so for sequence labeling tasks (Caruana, 1997; Collobert and Weston, 2008; Collobert et al., 2011). Results so far are not conclusive. Some works have reported negative results overall. For instance in their pioneering work, Collobert et al. (2011) observed that training their model on NER, POS tagging and chunking alto- gether led to slight decrease in performance com- pared to a similar model trained on each task sepa- rately. Søgaard and Goldberg (2016) study chunk- ing and CCG super tagging, coupled with an ad- ditional POS tagging task. They do report gains on both target tasks over single task models, but results varied depending where the additional task was taken care of in their architecture. The au- thors actually reported a failure to leverage other labelling tasks, and concluded that combined tasks …

Content truncated to 500 words