Semi-supervised sequence tagging with bidirectional language models

Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 1756–1765 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1161 Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 1756–1765 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1161 Semi-supervised sequence tagging with bidirectional language models Matthew E. Peters, Waleed Ammar, Chandra Bhagavatula, Russell Power Allen Institute for Artificial Intelligence {matthewp,waleeda,chandrab,russellp}@allenai.org Abstract Pre-trained word embeddings learned from unlabeled text have become a stan- dard component of neural network archi- tectures for NLP tasks. However, in most cases, the recurrent network that oper- ates on word-level representations to pro- duce context sensitive representations is trained on relatively little labeled data. In this paper, we demonstrate a general semi-supervised approach for adding pre- trained context embeddings from bidi- rectional language models to NLP sys- tems and apply it to sequence labeling tasks. We evaluate our model on two stan- dard datasets for named entity recognition (NER) and chunking, and in both cases achieve state of the art results, surpassing previous systems that use other forms of transfer or joint learning with additional labeled data and task specific gazetteers. 1 Introduction Due to their simplicity and efficacy, pre-trained word embedding have become ubiquitous in NLP systems. Many prior studies have shown that they capture useful semantic and syntactic information (Mikolov et al., 2013; Pennington et al., 2014) and including them in NLP systems has been shown to be enormously helpful for a variety of downstream tasks (Collobert et al., 2011). However, in many NLP tasks it is essential to represent not just the meaning of a word, but also the word in context. For example, in the two phrases “A Central Bank spokesman” and “The Central African Republic”, the word ‘Central’ is used as part of both an Organization and Location. Accordingly, current state of the art sequence tag- ging models typically include a bidirectional re- current neural network (RNN) that encodes token sequences into a context sensitive representation before making token specific predictions (Yang et al., 2017; Ma and Hovy, 2016; Lample et al., 2016; Hashimoto et al., 2016). Although the token representation is initialized with pre-trained embeddings, the parameters of the bidirectional RNN are typically learned only on labeled data. Previous work has explored meth- ods for jointly learning the bidirectional RNN with supplemental labeled data from other tasks (e.g., Søgaard and Goldberg, 2016; Yang et al., 2017). In this paper, we explore an alternate semi- supervised approach which does not require ad- ditional labeled data. We use a neural language model (LM), pre-trained on a large, unlabeled cor- pus to compute an encoding of the context at each position in the sequence (hereafter an LM embed- ding) and use it in the supervised sequence tag- ging model. Since the LM embeddings are used to compute the probability of future words in a neu- ral LM, they are likely to encode both the semantic …

Content truncated to 500 words