Unsupervised Paraphrasing Consistency Training for Low Resource Named Entity Recognition

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5303–5308 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 5303 Unsupervised Paraphrasing Consistency Training for Low Resource Named Entity Recognition Rui Wang Duke University rw161@duke.edu Ricardo Henao Duke University ricardo.henao@duke.edu Abstract Unsupervised consistency training is a way of semi-supervised learning that encourages con- sistency in model predictions between the orig- inal and augmented data. For Named Entity Recognition (NER), existing approaches aug- ment the input sequence with token replace- ment, assuming annotations on the replaced positions unchanged. In this paper, we explore the use of paraphrasing as a more principled data augmentation scheme for NER unsuper- vised consistency training. Specifically, we convert Conditional Random Field (CRF) into a multi-label classification module and encour- age consistency on the entity appearance be- tween the original and paraphrased sequences. Experiments show that our method is espe- cially effective when annotations are limited. 1 Introduction Supervised training for Named Entity Recognition (NER) requires token level annotations, which are time consuming and more expensive to obtain than the sequence level annotations commonly used for classification tasks. Due to the scarcity of labeled data, various semi-supervised approaches have been investigated for training in low-resource scenarios, i.e., only a small amount of labeled data is available (Clark et al., 2018; Lowell et al., 2020). Unsupervised consistency training is a common approach to semi-supervised learning in NER. It en- courages prediction consistency between the origi- nal and the augmented examples, by leveraging the availability of a larger amount of unlabeled data. Recently, Xie et al. (2019) proposed the Unsuper- vised Data Augmentation (UDA), which substitutes traditional token-wise perturbations with higher quality data augmentation, e.g., paraphrasing via back-translation. UDA achieves state-of-the-art results on a wide variety of classification tasks with only tens or hundreds of labeled examples, and even sometimes matching the performance of supervised training with a much larger (fully- annotated) dataset. In the case of NER, due to the difficulty of obtaining token-level annotations, it is of interest to extend UDA for NER models whose predictions are (token-level) sequences instead of single (sentence-level) labels. More recently, Lowell et al. (2020) augmented unlabeled samples for NER by randomly replacing a portion of input tokens with outputs from a lan- guage model, thus constraining the model predic- tions to be invariant to the replacement operation. There are two problems with this approach: i) there is no guarantee that the type of entity (label) will remain unchanged after replacement; and ii) the newly generated context from replacement is con- strained by length of the original sequence, which restricts the quality of augmentation. In fact, Xie et al. (2019) suggests that there exists strong corre- lation between the quality of the augmentation and the performance of consistency training. In this paper, we explore the use of paraphrasing as a means for higher quality data augmentation for unsupervised consistency training in NER. Com- pared with token replacement, the key difficulty of using paraphrasing is that the alignment of tokens …

Content truncated to 500 words