Label-Agnostic Sequence Labeling by Copying Nearest Neighbors

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5363–5369 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 5363 Label-Agnostic Sequence Labeling by Copying Nearest Neighbors Sam Wiseman Karl Stratos Toyota Technological Institute at Chicago Chicago, IL, USA {swiseman,stratos}@ttic.edu Abstract Retrieve-and-edit based approaches to struc- tured prediction, where structures associated with retrieved neighbors are edited to form new structures, have recently attracted in- creased interest. However, much recent work merely conditions on retrieved structures (e.g., in a sequence-to-sequence framework), rather than explicitly manipulating them. We show we can perform accurate sequence labeling by explicitly (and only) copying labels from retrieved neighbors. Moreover, because this copying is label-agnostic, we can achieve im- pressive performance in zero-shot sequence- labeling tasks. We additionally consider a dy- namic programming approach to sequence la- beling in the presence of retrieved neighbors, which allows for controlling the number of distinct (copied) segments used to form a pre- diction, and leads to both more interpretable and accurate predictions. 1 Introduction Retrieve-and-edit style structured prediction, where a model retrieves a set of labeled nearest neighbors from the training data and conditions on them to generate the target structure, is a promising approach that has recently received renewed interest (Hashimoto et al., 2018; Guu et al., 2018; Gu et al., 2018; Weston et al., 2018). This approach captures the intuition that while generating a highly complex structure from scratch may be difficult, editing a sufficiently similar structure or set of structures may be easier. Recent work in this area primarily uses the near- est neighbors and their labels simply as an ad- ditional context for a sequence-to-sequence style model to condition on. While effective, these models may not explicitly capture the discrete op- erations (like copying) that allow for the neighbors to be edited into the target structure, making inter- preting the behavior of the model difficult. More- over, since many retrieve-and-edit style models condition on dataset-specific labels directly, they may not easily allow for transfer learning and in particular to porting a trained model to a new task with different labels. We address these limitations in the context of sequence labeling by developing a simple label- agnostic model that explicitly models copying token-level labels from retrieved neighbors. Since the model is not a function of the labels themselves but only of a learned notion of similarity between an input and retrieved neighbor inputs, it can be effortlessly ported (zero shot) to a task with differ- ent labels, without any retraining. Such a model can also take advantage of recent advances in rep- resentation learning, such as BERT (Devlin et al., 2018), in defining this similarity. We evaluate the proposed approach on stan- dard sequence labeling tasks, and show it is com- petitive with label-dependent approaches when trained on the same data, but substantially outper- forms strong baselines when it comes to zero-shot transfer applications, such as when training with coarse labels and testing with fine-grained labels. Finally, we propose …

Content truncated to 500 words