Consistency Training with Virtual Adversarial Discrete Perturbation

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5646 - 5656 July 10-15, 2022 ©2022 Association for Computational Linguistics Consistency Training with Virtual Adversarial Discrete Perturbation Jungsoo Park1,3∗ Gyuwan Kim2∗† Jaewoo Kang3 1Clova AI, NAVER Corp. 2 University of California, Santa Barbara 3Korea University {jungsoo_park,kangj}@korea.ac.kr gyuwankim@ucsb.edu Abstract Consistency training regularizes a model by enforcing predictions of original and perturbed inputs to be similar. Previous studies have pro- posed various augmentation methods for the perturbation but are limited in that they are ag- nostic to the training model. Thus, the per- turbed samples may not aid in regularization due to their ease of classification from the model. In this context, we propose an aug- mentation method of adding a discrete noise that would incur the highest divergence be- tween predictions. This virtual adversarial dis- crete noise obtained by replacing a small por- tion of tokens while keeping original seman- tics as much as possible efficiently pushes a training model’s decision boundary. Experi- mental results show that our proposed method outperforms other consistency training base- lines with text editing, paraphrasing, or a con- tinuous noise on semi-supervised text classifi- cation tasks and a robustness benchmark1. 1 Introduction Building a natural language processing (NLP) sys- tem often requires an expensive process to col- lect a massive amount of labeled text data. Semi- supervised learning (SSL) (Chapelle et al., 2009) mitigates the requirement for such labeled data by exploiting the structure of unlabeled data. Among the SSL methods, the consistency training frame- work (Laine and Aila, 2017; Sajjadi et al., 2016) enforces a model to produce similar predictions of original and perturbed inputs. This method has several advantages over other training algorithms such as naively adding augmented samples into the training set (Wei and Zou, 2019; Ng et al., 2020) in that it provides a richer training signal than a one-hot label, and also applies to both labeled and unlabeled data (Xie et al., 2020). ∗Equal contribution. †Work done while working at NAVER Clova & AI Lab 1Code repo: https://github.com/clovaai/vat-d (a) Real data distribution, which requires complex de- cision boundary. (b) A simple decision boundary is drawn when samples are insufficient. (c) Augmentations can push the decision boundary (dot- ted line) from the current one (bold line). (d) Augmentations which are outside the current de- cision boundary enable fur- ther pushing it. Figure 1: A simple illustration of the intuition behind our method is visualized in the two-dimensional space, where the augmented samples (triangle) would aid the training given a limited number of data (circle). For perturbing a text while preserving its seman- tics, some approaches inject continuous noise to embedding vectors (Xie et al., 2017; Miyato et al., 2018), and others modify text itself in discrete fash- ion by edit operations (Kobayashi, 2018; Wei and Zou, 2019) or paraphrasing with back-translation (Sennrich et al., 2016; Edunov et al., 2018; Xie et al., 2020). However, adding continuous noise might not strongly regularize …

Content truncated to 500 words