How Effective is Task-Agnostic Data Augmentation for Pretrained Transformers?

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4401–4411 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 4401 How Effective is Task-Agnostic Data Augmentation for Pretrained Transformers? Shayne Longpre∗ Apple Inc. slongpre@apple.com Yu Wang∗ Apple Inc. w.y@apple.com Christopher DuBois Apple Inc. cdubois@apple.com Abstract Task-agnostic forms of data augmentation have proven widely effective in computer vi- sion, even on pretrained models. In NLP simi- lar results are reported most commonly for low data regimes, non-pretrained models, or situa- tionally for pretrained models. In this paper we ask how effective these techniques really are when applied to pretrained transformers. Using two popular varieties of task-agnostic data augmentation (not tailored to any partic- ular task), Easy Data Augmentation (Wei and Zou, 2019) and Back-Translation (Sennrich et al., 2015), we conduct a systematic exam- ination of their effects across 5 classification tasks, 6 datasets, and 3 variants of modern pretrained transformers, including BERT, XL- NET, and ROBERTA. We observe a nega- tive result, finding that techniques which pre- viously reported strong improvements for non- pretrained models fail to consistently improve performance for pretrained transformers, even when training data is limited. We hope this empirical analysis helps inform practitioners where data augmentation techniques may con- fer improvements. 1 Introduction “Task-agnostic” data augmentations — those which are not tailored to a task, but are broadly applicable across the visual or textual domain — have long been a staple of machine learning. Task-agnostic data augmentation techniques for computer vision, such as image translation, rotation, shearing, and contrast jittering, have achieved considerable suc- cess, given their ease of use, and wide-spread appli- cability (Cubuk et al., 2018; Perez and Wang, 2017). In natural language processing, benefits of data aug- mentation have usually been observed where the augmentations are suited to the task: as with back- translation for machine translation (Edunov et al., ∗ equal contribution 2018; Xia et al., 2019), or negative sampling for question answering and document retrieval (Zhang et al., 2017; Yang et al., 2019a; Xiong et al., 2020). Outside of application-tailored augmentations, im- provements are primarily reported on autoregres- sive models without unsupervised pretraining or contextual embeddings, such as LSTMs and CNNs, and even then in low data regimes (Zhang et al., 2015; Coulombe, 2018; Wei and Zou, 2019; Yu et al., 2018). Additionally, in computer vision task- agnostic augmentations continue to report bene- fits when applied to pretrained representations (Gu et al., 2019). However, in NLP it is less clear whether these general augmentations benefit mod- ern Transformer architectures with unsupervised pretraining at scale. We pose the question: to what extent do mod- ern NLP models benefit from task-agnostic data augmentations? In this paper, we provide empir- ical results across a variety of tasks, datasets, ar- chitectures, and popular augmentation strategies. Among data augmentation techniques, we select Easy Data Augmentation (Wei and Zou, 2019) and Back-Translation (Sennrich et al., 2015); EDA and BT respectively. Both are popular task-agnostic options, and report significant gains for LSTMs on a wide variety of …

Content truncated to 500 words