Unsupervised Data Augmentation with Naive Augmentation and without Unlabeled Data

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4992–5001 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 4992 Unsupervised Data Augmentation with Naive Augmentation and without Unlabeled Data David Lowell Northeastern University lowell.d@husky.neu.edu Brian E Howard Sciome LLC brian.howard@sciome.com Zachary C. Lipton Carnegie Mellon University zlipton@cmu.edu Byron C. Wallace Northeastern University b.wallace@northeastern.edu Abstract Unsupervised Data Augmentation (UDA) is a semi-supervised technique that applies a consistency loss to penalize differences be- tween a model’s predictions on (a) observed (unlabeled) examples; and (b) corresponding ‘noised’ examples produced via data augmen- tation. While UDA has gained popularity for text classification, open questions linger over which of its components are important, and how to extend the method to sequence label- ing tasks; this paper addresses these ques- tions. Our main contribution is an empiri- cal study of UDA to establish which com- ponents of the algorithm confer benefits in NLP. Notably, although prior work has empha- sized use of clever augmentation techniques in- cluding back-translation, we find that enforc- ing consistency between predictions assigned to observed and randomly substituted words often yields comparable (or greater) benefits compared to these more complex perturba- tion models. Furthermore, we find that apply- ing UDA’s consistency loss affords meaningful gains without any unlabeled data at all, i.e., in a standard supervised setting. In short, UDA need not be unsupervised to realize much of its noted benefits, and does not require com- plex data augmentation to be effective. 1 Introduction While the advent of large neural models has led to rapid progress on a wide spectrum of prediction benchmarks in NLP, these methods tend to require large amounts of training data. This limitation is particularly acute in domains such as information extraction from scientific documents, where unla- beled in-domain data is plentiful but labeled data is rare and requires significant annotator experience to produce. The cost of acquiring data in such do- mains has spurred interest in developing models that can achieve greater extraction accuracy, even when the available labelled corpora are small (Nye et al., 2018; Maharana et al., 2018). In this paper, we investigate Unsupervised Data Augmentation (UDA; Xie et al. 2019), a re- cently proposed semi-supervised learning method in which models are trained on both labeled and unlabeled in-domain data. The learning objective for the unlabeled component entails minimizing the divergence between the model’s outputs on a given example and its outputs on a perturbed ver- sion of the same example. While this combination of data augmentation with a consistency loss was previously proposed as Invariant Representation Learning and demonstrated utility in the context of speech recognition (Liang et al., 2018), UDA applies the method in a semi-supervised setting, in a manner similar to virtual adversarial training (Miyato et al., 2018). The original UDA paper (Xie et al., 2019) reported significant benefits on both computer vision and NLP tasks. Producing such perturbed examples requires specifying a data augmentation pipeline. Typically, these apply one or more transformations that (the practitioner hopes) do …

Content truncated to 500 words