Style Transfer as Data Augmentation: A Case Study on Named Entity Recognition

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1827 - 1841 December 7-11, 2022 ©2022 Association for Computational Linguistics Style Transfer as Data Augmentation: A Case Study on Named Entity Recognition Shuguang Chen University of Houston schen52@uh.edu Leonardo Neves Snap Inc. lneves@snap.com Thamar Solorio University of Houston tsolorio@uh.edu Abstract In this work, we take the named entity recog- nition task in the English language as a case study and explore style transfer as a data aug- mentation method to increase the size and di- versity of training data in low-resource sce- narios. We propose a new method to effec- tively transform the text from a high-resource domain to a low-resource domain by chang- ing its style-related attributes to generate syn- thetic data for training. Moreover, we de- sign a constrained decoding algorithm along with a set of key ingredients for data selec- tion to guarantee the generation of valid and coherent data. Experiments and analysis on five different domain pairs under different data regimes demonstrate that our approach can significantly improve results compared to cur- rent state-of-the-art data augmentation meth- ods. Our approach is a practical solution to data scarcity, and we expect it to be applicable to other NLP tasks. 1 1 Introduction Large-scale pre-trained language models (PLMs) such as BERT (Devlin et al., 2019) and T5 (Raffel et al., 2020) have shown impressive performances on a wide variety of NLP tasks. Following the paradigm of self-supervised pre-training and fine- tuning, these models have achieved state-of-the- art performance in many NLP benchmarks such as question answering (Yang et al., 2019; Yamada et al., 2020), machine translation (Provilkov et al., 2020; Lewis et al., 2020), and text summariza- tion (Zaheer et al., 2020; Aghajanyan et al., 2021). However, due to the discrepancy of training objec- tives between language modeling and downstream tasks, the performance of such models may be limited by data scarcity in low-resource domains (Jiang et al., 2020; Gururangan et al., 2020). 1We released the code at https://github.com/ RiTUAL-UH/DA_NER. Data augmentation is effective in addressing data scarcity. Previous work (Wei and Zou, 2019; Morris et al., 2020) has mainly focused on us- ing in-domain data to synthesize new datasets for training. When applied to low-resource domains, however, these approaches may not lead to sig- nificant improvement gains as the data in low- resource domains is not comparable to that in high-resource domains in terms of size and diver- sity. Recently, many studies (Xia et al., 2019; De- houck and Gómez-Rodríguez, 2020) have revealed the potential of leveraging data from high-resource domains to improve low-resource tasks. Despite their impressive results, directly using abundant data from high-resource domains can be problem- atic due to the difference in data distribution (e.g., language shift) and feature misalignment (e.g., class mismatch) between domains (Wang et al., 2018; Zhang et al., 2021). In this work, we explore the potential of em- ploying style transfer as a way of data augmen- tation in cross-domain settings. Style transfer on natural language …

Content truncated to 500 words