Learning to Perturb Word Embeddings for Out-of-distribution QA

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 5583–5595 August 1–6, 2021. ©2021 Association for Computational Linguistics 5583 Learning to Perturb Word Embeddings for Out-of-distribution QA Seanie Lee1∗ Minki Kang1∗ Juho Lee1, Sung Ju Hwang1,2 KAIST1, AITRICS2, South Korea {lsnfamily02, zzx1133, juholee, sjhwang82}@kaist.ac.kr Abstract QA models based on pretrained language mod- els have achieved remarkable performance on various benchmark datasets. However, QA models do not generalize well to unseen data that falls outside the training distribution, due to distributional shifts. Data augmentation (DA) techniques which drop/replace words have shown to be effective in regularizing the model from overfitting to the training data. Yet, they may adversely affect the QA tasks since they incur semantic changes that may lead to wrong answers for the QA task. To tackle this problem, we propose a simple yet effective DA method based on a stochastic noise generator, which learns to perturb the word embedding of the input questions and context without changing their semantics. We validate the performance of the QA models trained with our word embedding perturbation on a single source dataset, on five different target domains. The results show that our method significantly outperforms the baseline DA methods. Notably, the model trained with ours outperforms the model trained with more than 240K artificially generated QA pairs. 1 Introduction Deep learning models have achieved impressive performances on a variety of real-world natural lan- guage understanding tasks such as text classifica- tion, machine translation, question answering, and text generation to name a few (Vaswani et al., 2017; Seo et al., 2017). Recently, language models that are pretrained with a large amount of unlabeled data have achieved breakthrough in the performance on these downstream tasks (Devlin et al., 2019), even surpassing human performance on some of them. The success of such data-driven language model pretraining heavily depends on the amount and diversity of training data available, since when ∗* Equal contribution trained with a small amount of highly-biased data, the pretrained models can overfit and may not generalize well to out-of-distribution data. Data augmentation (DA) techniques (Krizhevsky et al., 2012; Verma et al., 2019a; Yun et al., 2019; Sen- nrich et al., 2016) can prevent this to a certain ex- tent, but most of them are developed for image domains and are not directly applicable to augment- ing words and texts. Perhaps the most important desiderata for an augmentation method in super- vised learning, is that it should not change the label of an example. For image domains, there exist several well-defined data augmentation techniques that can produce diverse augmented images with- out changing the semantics. In contrast, for Natural Language Processing (NLP), it is not straightfor- ward to augment the input texts without changing their semantics. A simple augmentation technique that preserves semantics is replacing words with synonyms or using back translation (Sennrich et al., 2016). However, they do not effectively improve the generalization performance because the diver- sity …

Content truncated to 500 words