Textual Data Augmentation for Efficient Active Learning on Tiny Datasets

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 7400–7410, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 7400 Textual Data Augmentation for Efficient Active Learning on Tiny Datasets Husam Quteineh1∗, Spyridon Samothrakis2, and Richard Sutcliffe3∗ 1Business and Local Government Data Research Centre, School of CSEE, University of Essex husam.quteineh@essex.ac.uk 2IADS, University of Essex ssamot@essex.ac.uk 3IPMI Group, School of IST, Northwest University China rsutcl@nwu.edu.cn Abstract In this paper we propose a novel data augmen- tation approach where guided outputs of a lan- guage generation model, e.g. GPT-2, when la- beled, can improve the performance of text classifiers through an active learning process. We transform the data generation task into an optimization problem which maximizes the usefulness of the generated output, using Monte Carlo Tree Search (MCTS) as the opti- mization strategy and incorporating entropy as one of the optimization criteria. We test our approach against a Non-Guided Data Genera- tion (NGDG) process that does not optimize for a reward function. Starting with a small set of data, our results show an increased per- formance with MCTS of 26% on the TREC- 6 Questions dataset, and 10% on the Stanford Sentiment Treebank SST-2 dataset. Compared with NGDG, we are able to achieve increases of 3% and 5% on TREC-6 and SST-2. 1 Introduction Active learning (AL) is a well applied approach in areas where unlabeled data is abundantly available, but labels are either scarce or costly to obtain (Set- tles, 2009). In AL, a classifier is improved upon through an iterative learning process; at each cycle, a subset of the original dataset with the most infor- mative examples is selected to be labeled, typically by a human expert, before it is then added to the existing training data (Settles, 2009). Previous active learning research on textual data to our knowledge has always assumed the availabil- ity of datasets containing large pools of unlabeled data. In cases where the available data is insuffi- cient for active learning, the burden is transferred to the data collection process, where additional ∗ Corresponding author data must either be manually created, or collected from real world interactions. One strategy for cre- ating data is to transform existing examples in cer- tain ways in order to produce new data items and hence increase the size of the training dataset. This approach has been applied successfully in com- puter vision for example, by manipulating exist- ing images while preserving the label to create additional data points (Shorten and Khoshgoftaar, 2019). However, in NLP, augmenting data is a very difficult task due to the complex nature of language (Wei and Zou, 2019). In this paper we assume a real-life scenario where the data at hand is insufficient for running a typical active learning algorithm. We introduce a method that enables us to automatically generate artificial text examples that complement an existing dataset. Our approach minimizes the human factor in data creation by au- tomating the process through a guided searching procedure. Once a set of …

Content truncated to 500 words