Zero-Shot Sequence Labeling: Transferring Knowledge from Sentences to Tokens

Proceedings of NAACL-HLT 2018, pages 293–302 New Orleans, Louisiana, June 1 - 6, 2018. c⃝2018 Association for Computational Linguistics Zero-shot Sequence Labeling: Transferring Knowledge from Sentences to Tokens Marek Rei The ALTA Institute Computer Laboratory University of Cambridge United Kingdom marek.rei@cl.cam.ac.uk Anders Søgaard CoAStaL DIKU Department of Computer Science University of Copenhagen Denmark soegaard@di.ku.dk Abstract Can attention- or gradient-based visualization techniques be used to infer token-level labels for binary sequence tagging problems, using networks trained only on sentence-level la- bels? We construct a neural network architec- ture based on soft attention, train it as a binary sentence classifier and evaluate against token- level annotation on four different datasets. In- ferring token labels from a network provides a method for quantitatively evaluating what the model is learning, along with generating use- ful feedback in assistance systems. Our results indicate that attention-based methods are able to predict token-level labels more accurately, compared to gradient-based methods, some- times even rivaling the supervised oracle net- work. 1 Introduction Sequence labeling is a structured prediction task where systems need to assign the correct label to every token in the input sequence. Many NLP tasks, including part-of-speech tagging, named entity recognition, chunking, and error detec- tion, are often formulated as variations of se- quence labeling. Recent state-of-the-art models make use of bidirectional LSTM architectures (Ir- soy and Cardie, 2014), character-based represen- tations (Lample et al., 2016), and additional ex- ternal features (Peters et al., 2017). Optimiza- tion of these models requires appropriate training data where individual tokens are manually labeled, which can be time-consuming and expensive to obtain for each different task, domain and target language. In this paper, we investigate the task of per- forming sequence labeling without having access to any training data with token-level annotation. Instead of training the model directly to predict the label for each token, the model is optimized using a sentence-level objective and a modified version of the attention mechanism is then used to infer labels for individual words. While this approach is not expected to outper- form a fully supervised sequence labeling method, it opens possibilities for making use of text classi- fication datasets where collecting token-level an- notation is not possible or cost-effective. Inferring token-level labels from a text classi- fication network also provides a method for ana- lyzing and interpreting the model. Previous work has used attention weights to visualize the focus of neural models in the input data. However, these analyses have largely been qualitative examina- tions, looking at only a few examples from the datasets. By formulating the task as a zero-shot la- beling problem, we can provide quantitative eval- uations of what the model is learning and where it is focusing. This will allow us to measure whether the features that the model is learning actually match our intuition, provide informative feedback to end-users, and guide our development of future model architectures. 2 Network Architecture The main system takes as input a sentence, sep- arated into tokens, and outputs a binary predic- tion …

Content truncated to 500 words