A Data Cartography based MixUp for Pre-trained Language Models

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4244 - 4250 July 10-15, 2022 ©2022 Association for Computational Linguistics A Data Cartography based MixUp for Pre-trained Language Models Seo Yeon Park and Cornelia Caragea Computer Science University of Illinois Chicago spark313@uic.edu cornelia@uic.edu Abstract MixUp is a data augmentation strategy where additional samples are generated during train- ing by combining random pairs of training samples and their labels. However, select- ing random pairs is not potentially an opti- mal choice. In this work, we propose TD- MixUp, a novel MixUp strategy that lever- ages Training Dynamics and allows more in- formative samples to be combined for gener- ating new data samples. Our proposed TD- MixUp first measures confidence, variability, (Swayamdipta et al., 2020), and Area Under the Margin (AUM) (Pleiss et al., 2020) to iden- tify the characteristics of training samples (e.g., as easy-to-learn or ambiguous samples), and then interpolates these characterized samples. We empirically validate that our method not only achieves competitive performance using a smaller subset of the training data compared with strong baselines, but also yields lower ex- pected calibration error on the pre-trained lan- guage model, BERT, on both in-domain and out-of-domain settings in a wide range of NLP tasks. We publicly release our code.1 1 Introduction MixUp (Zhang et al., 2018) is a simple data aug- mentation strategy in which additional samples are generated during training by combining random pairs of training samples and their labels. While simple to implement, MixUp has been shown to improve both predictive performance and model calibration (i.e., avoiding over-confident predic- tions) (Guo et al., 2017) due to its regularization effect through data augmentation (Thulasidasan et al., 2019). However, selecting random pairs in MixUp might not necessarily be optimal. Despite this, MixUp has been explored for NLP tasks with substantial success using hidden state representations (Verma et al., 2019). For instance, Sun et al. (2020) explored MixUp, which uses the 1https://github.com/seoyeon-p/TDMixUp hidden representation of BERT (Devlin et al., 2019) to synthesize additional samples from randomly se- lected pairs. Yin et al. (2021) proposed MixUp, which uses the hidden representation of RoBERTa (Liu et al., 2019) to interpolate all samples in the same mini-batch to better cover the feature space. To date, only a few prior works have focused on se- lecting informative samples for MixUp. For exam- ple, Chen et al. (2020) proposed semi-supervised learning, which interpolates labeled and unlabeled data based on entropy. Kong et al. (2020) explored BERT calibration with MixUp, which generates new samples by exploiting the distance between samples in the feature space. Recently, Swayamdipta et al. (2020) introduced data maps, which allow evaluating data quality by using training dynamics (i.e., the behavior of a model as training progresses). Specifically, they consider the mean and standard deviation of the gold label probabilities, predicted for each sample across training epochs (i.e., confidence and vari- ability), and characterize data into three different categories: (1) samples that the …

Content truncated to 500 words