Semi-supervised Multitask Learning for Sequence Labeling

Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 2121–2130 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1194 Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 2121–2130 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1194 Semi-supervised Multitask Learning for Sequence Labeling Marek Rei The ALTA Institute Computer Laboratory University of Cambridge United Kingdom marek.rei@cl.cam.ac.uk Abstract We propose a sequence labeling frame- work with a secondary training objec- tive, learning to predict surrounding words for every word in the dataset. This lan- guage modeling objective incentivises the system to learn general-purpose patterns of semantic and syntactic composition, which are also useful for improving accu- racy on different sequence labeling tasks. The architecture was evaluated on a range of datasets, covering the tasks of error detection in learner texts, named entity recognition, chunking and POS-tagging. The novel language modeling objective provided consistent performance improve- ments on every benchmark, without re- quiring any additional annotated or unan- notated data. 1 Introduction Accurate and efficient sequence labeling mod- els have a wide range of applications, including named entity recognition (NER), part-of-speech (POS) tagging, error detection and shallow pars- ing. Specialised approaches to sequence label- ing often include extensive feature engineering, such as integrated gazetteers, capitalisation fea- tures, morphological information and POS tags. However, recent work has shown that neural net- work architectures are able to achieve compara- ble or improved performance, while automatically discovering useful features for a specific task and only requiring a sequence of tokens as input (Col- lobert et al., 2011; Irsoy and Cardie, 2014; Lample et al., 2016). This feature discovery is usually driven by an objective function based on predicting the anno- tated labels for each word, without much incentive to learn more general language features from the available text. In many sequence labeling tasks, the relevant labels in the dataset are very sparse and most of the words contribute very little to the training process. For example, in the CoNLL 2003 NER dataset (Tjong Kim Sang and De Meulder, 2003) only 17% of the tokens represent an entity. This ratio is even lower for error detection, with only 14% of all tokens being annotated as an error in the FCE dataset (Yannakoudakis et al., 2011). The sequence labeling models are able to learn this bias in the label distribution without obtaining much additional information from words that have the majority label (O for outside of an entity; C for correct word). Therefore, we propose an addi- tional training objective which allows the models to make more extensive use of the available data. The task of language modeling offers an eas- ily accessible objective – learning to predict the next word in the sequence requires only plain text as input, without relying on any particular annota- tion. Neural language modeling architectures also have many similarities to common sequence label- ing frameworks: words are first mapped to dis- tributed …

Content truncated to 500 words