Learning to Generate Examples for Semantic Processing Tasks

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4587 - 4601 July 10-15, 2022 ©2022 Association for Computational Linguistics Learning to Generate Examples for Semantic Processing Tasks Danilo Croce Dept. of Enterprise Engineering University of Rome, Tor Vergata Roma, Italy croce@info.uniroma2.it Simone Filice Amazon Tel Aviv, Israel filicesf@amazon.com Giuseppe Castellucci Amazon Seattle, USA giusecas@amazon.com Roberto Basili Dept. of Enterprise Engineering University of Rome, Tor Vergata Roma, Italy basili@info.uniroma2.it Abstract Even if recent Transformer-based architec- tures, such as BERT, achieved impressive re- sults in semantic processing tasks, their fine- tuning stage still requires large scale training resources. Usually, Data Augmentation (DA) techniques can help to deal with low resource settings. In Text Classification tasks, the ob- jective of DA is the generation of well-formed sentences that (i) represent the desired task cat- egory and (ii) are novel with respect to exist- ing sentences. In this paper, we propose a neu- ral approach to automatically learn to generate new examples using a pre-trained sequence-to- sequence model. We first learn a task-oriented similarity function that we use to pair simi- lar examples. Then, we use these example pairs to train a model to generate examples. Experiments in low resource settings show that augmenting the training material with the proposed strategy systematically improves the results on text classification and natural lan- guage inference tasks by up to 10% accuracy, outperforming existing DA approaches. 1 Introduction Deep Learning models achieve state-of-the-art re- sults in many domains, including Computer Vision and Natural Language Processing (NLP). Training these large models typically requires many exam- ples, whose collection and annotation can be costly and time-consuming. Data augmentation (DA) has proven an efficient way to acquire more training samples without incurring in the prohibitive anno- tation cost in a variety of fields, including com- puter vision (Perez and Wang, 2017) and speech recognition (Rebai et al., 2017). In NLP, some DA techniques have been proposed too, as surveyed in (Feng et al., 2021): common approaches cre- ate synthetic data by manipulating real examples, using Text Editing (Wei and Zou, 2019) or Back- Translation (Sennrich et al., 2015); the resulting examples are automatically labeled by inheriting the class of the original example they were gener- ated from. Unfortunately, when using recent pre-trained lan- guage models, such as BERT (Devlin et al., 2018) or RoBERTa (Liu et al., 2019), the effectiveness of DA methods is extremely limited, and some- times they can even hurt the results (Longpre et al., 2020). A possible explanation for this inefficacy is that these DA techniques introduce lexical and structural variability that accurate language models directly induce during pre-training. The usefulness of synthetic examples is strictly related to their di- versity from the original training data. At the same time, diverging too much from the initial data might increase the risk of introducing noisy annotations, i.e., synthetic data not reflecting the class of the original examples they were generated from. To directly tackle the …

Content truncated to 500 words