SeqVAT: Virtual Adversarial Training for Semi-Supervised Sequence Labeling

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8801–8811 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 8801 SeqVAT: Virtual Adversarial Training for Semi-Supervised Sequence Labeling Luoxin Chen, Weitong Ruan, Xinyue Liu, Jianhua Lu Amazon Alexa AI, Cambridge, MA, US {luoxchen, weiton, luxnyu, jianhual}@amazon.com Abstract Virtual adversarial training (VAT) is a power- ful technique to improve model robustness in both supervised and semi-supervised settings. It is effective and can be easily adopted on lots of image classification and text classifica- tion tasks. However, its benefits to sequence labeling tasks such as named entity recogni- tion (NER) have not been shown as signifi- cant, mostly, because the previous approach can not combine VAT with the conditional ran- dom field (CRF). CRF can significantly boost accuracy for sequence models by putting con- straints on label transitions, which makes it an essential component in most state-of-the- art sequence labeling model architectures. In this paper, we propose SeqVAT, a method which naturally applies VAT to sequence label- ing models with CRF. Empirical studies show that SeqVAT not only significantly improves the sequence labeling performance over base- lines under supervised settings, but also outper- forms state-of-the-art approaches under semi- supervised settings. 1 Introduction While having achieved great success on various computer vision and natural language processing tasks, deep neural networks, even state-of-the-art models, are usually vulnerable to tiny input pertur- bations (Szegedy et al., 2014; Goodfellow et al., 2015). To improve the model robustness against perturbations, Goodfellow et al. (2015) proposed to train neural networks on both original training examples and adversarial examples (examples gen- erated by adding small but worst-case perturbations to the original examples). This approach, named adversarial training (AT), has been reported to be highly effective on image classification (Goodfel- low et al., 2015), text classification (Miyato et al., 2017), as well as sequence labeling (Yasunaga et al., 2018). However, AT is limited to a supervised scenario, which uses the labels to compute adversarial losses. To make use of unlabeled data, virtual adversar- ial training (VAT) was proposed to extend AT to semi-supervised settings (Miyato et al., 2019). Un- like AT which treats adversarial examples as new training instances that have the same labels as orig- inal examples, VAT minimizes the KL divergence between estimated label distribution of original ex- amples and that of adversarial examples. In this manner, both labeled and unlabeled data can be used in training to improve accuracy and robust- ness. As a semi-supervised learning algorithm, VAT was reported to be effective on both image (Goodfellow et al., 2015; Miyato et al., 2019) and text classifications (Miyato et al., 2017). Moreover, a recent study (Oliver et al., 2018) conducted com- prehensive comparisons on various popular semi- supervised learning algorithms. VAT turned out to be the most effective one. Despite its success in classification tasks, VAT has not shown similar effectiveness in sequence la- beling tasks. In the conventional classification task, the model learns a mapping between a sentence (sequence of tokens) …

Content truncated to 500 words