GAN-BERT: Generative Adversarial Learning for Robust Text Classification with a Bunch of Labeled Examples

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2114–2119 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 2114 GAN-BERT: Generative Adversarial Learning for Robust Text Classification with a Bunch of Labeled Examples Danilo Croce Dept. of Enterprise Engineering University of Rome, Tor Vergata Roma, Italy croce@info.uniroma2.it Giuseppe Castellucci Amazon Seattle, USA giusecas@amazon.com Roberto Basili Dept. of Enterprise Engineering University of Rome, Tor Vergata Roma, Italy basili@info.uniroma2.it Abstract Recent Transformer-based architectures, e.g., BERT, provide impressive results in many Nat- ural Language Processing tasks. However, most of the adopted benchmarks are made of (sometimes hundreds of) thousands of exam- ples. In many real scenarios, obtaining high- quality annotated data is expensive and time- consuming; in contrast, unlabeled examples characterizing the target task can be, in gen- eral, easily collected. One promising method to enable semi-supervised learning has been proposed in image processing, based on Semi- Supervised Generative Adversarial Networks. In this paper, we propose GAN-BERT that ex- tends the fine-tuning of BERT-like architec- tures with unlabeled data in a generative adver- sarial setting. Experimental results show that the requirement for annotated examples can be drastically reduced (up to only 50-100 an- notated examples), still obtaining good perfor- mances in several sentence classification tasks. 1 Introduction In recent years, Deep Learning methods have be- come very popular in Natural Language Process- ing (NLP), e.g., they reach high performances by relying on very simple input representations (for example, in (Kim, 2014; Goldberg, 2016; Kim et al., 2016)). In particular, Transformer-based architectures, e.g., BERT (Devlin et al., 2019), pro- vide representations of their inputs as a result of a pre-training stage. These are, in fact, trained over large scale corpora and then effectively fine- tuned over a targeted task achieving state-of-the-art results in different and heterogeneous NLP tasks. These achievements are obtained when thousands of annotated examples exist for the final tasks. As experimented in this work, the quality of BERT fine-tuned over less than 200 annotated instances shows significant drops, especially in classification tasks involving many categories. Unfortunately, obtaining annotated data is a time-consuming and costly process. A viable solution is adopting semi- supervised methods, such as in (Weston et al., 2008; Chapelle et al., 2010; Yang et al., 2016; Kipf and Welling, 2016) to improve the generalization capa- bility when few annotated data is available, while the acquisition of unlabeled sources is possible. One effective semi-supervised method is imple- mented within Semi-Supervised Generative Adver- sarial Networks (SS-GANs). Usually, in GANs (Goodfellow et al., 2014) a “generator” is trained to produce samples resembling some data distribu- tion. This training process “adversarially” depends on a “discriminator”, which is instead trained to distinguish samples of the generator from the real instances. SS-GANs (Salimans et al., 2016) are an extension to GANs where the discriminator also assigns a category to each example while discrim- inating whether it was automatically generated or not. In SS-GANs, the labeled material is thus used to train the discriminator, while the unlabeled exam- ples (as …

Content truncated to 500 words