Semi-supervised Structured Prediction with Neural CRF Autoencoder

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1701–1711 Copenhagen, Denmark, September 7–11, 2017. c⃝2017 Association for Computational Linguistics Semi-Supervised Structured Prediction with Neural CRF Autoencoder Xiao Zhang†, Yong Jiang‡, Hao Peng†, Kewei Tu‡ and Dan Goldwasser† †Department of Computer Science, Purdue University, West Lafayette, USA {zhang923, pengh, dgoldwas}@cs.purdue.edu ‡School of Information Science and Technology, ShanghaiTech University, Shanghai, China {jiangyong, tukw}@shanghaitech.edu.cn Abstract In this paper we propose an end-to- end neural CRF autoencoder (NCRF-AE) model for semi-supervised learning of se- quential structured prediction problems. Our NCRF-AE consists of two parts: an encoder which is a CRF model enhanced by deep neural networks, and a decoder which is a generative model trying to re- construct the input. Our model has a uni- fied structure with different loss functions for labeled and unlabeled data with shared parameters. We developed a variation of the EM algorithm for optimizing both the encoder and the decoder simultaneously by decoupling their parameters. Our ex- perimental results over the Part-of-Speech (POS) tagging task on eight different lan- guages, show that the NCRF-AE model can outperform competitive systems in both supervised and semi-supervised sce- narios. 1 Introduction The recent renaissance of deep learning has led to significant strides forward in several AI fields. In Natural Language Processing (NLP), charac- terized by highly structured tasks, promising re- sults were obtained by models that combine deep learning methods with traditional structured learn- ing algorithms (Chen and Manning, 2014; Dur- rett and Klein, 2015; Andor et al., 2016; Wise- man and Rush, 2016). These models combine the strengths of neural models, that can score local decisions using a rich non-linear representation, with efficient inference procedures used to com- bine the local decisions into a coherent global de- cision. Among these models, neural variants of the Conditional Random Fields (CRF) model (Laf- ferty et al., 2001) are especially popular. By re- placing the linear potentials with non-linear poten- tial using neural networks these models were able to improve performance in several structured pre- diction tasks (Andor et al., 2016; Peng and Dredze, 2016; Lample et al., 2016; Ma and Hovy, 2016; Durrett and Klein, 2015). Despite their promise, wider adoption of these algorithms for new structured prediction tasks can be difficult. Neural networks are notoriously sus- ceptible to over-fitting unless large amounts of training data are available. This problem is exacer- bated in the structured settings, as accounting for the dependencies between decisions requires even more data. Providing it through manual annotation is often a difficult labor-intensive task. In this paper we tackle this problem, and propose an end-to-end neural CRF autoencoder (NCRF-AE) model for semi-supervised learning on sequence labeling problems. An autoencoder is a special type of neural net, modeling the conditional probability P( ˆX|X), where X is the original input to the model and ˆX is the reconstructed input (Hinton and Zemel, 1994). Autoencoders consist of two parts, an en- coder projecting the input to a hidden space, and a decoder reconstructing the input from …

Content truncated to 500 words