Adversarial Training for Machine Reading Comprehension with Virtual Embeddings

Proceedings of the 10th Conference on Lexical and Computational Semantics, pages 308–313 August 5–6, 2021, Bangkok, Thailand (online) ©2021 Association for Computational Linguistics 308 Adversarial Training for Machine Reading Comprehension with Virtual Embeddings Ziqing Yang1†, Yiming Cui2,1†, Chenglei Si3,Wanxiang Che2, Ting Liu2, Shijin Wang1,Guoping Hu1 1State Key Laboratory of Cognitive Intelligence, iFLYTEK Research, China 2 Research Center for SCIR, Harbin Institute of Technology, Harbin, China 3 University of Maryland, College Park MD, USA †{zqyang5,ymcui}@iflytek.com Abstract Adversarial training (AT) as a regularization method has proved its effectiveness on vari- ous tasks. Though there are successful appli- cations of AT on some NLP tasks, the distin- guishing characteristics of NLP tasks have not been exploited. In this paper, we aim to apply AT on machine reading comprehension (MRC) tasks. Furthermore, we adapt AT for MRC tasks by proposing a novel adversarial training method called PQAT that perturbs the embed- ding matrix instead of word vectors. To dif- ferentiate the roles of passages and questions, PQAT uses additional virtual P/Q-embedding matrices to gather the global perturbations of words from passages and questions separately. We test the method on a wide range of MRC tasks, including span-based extractive RC and multiple-choice RC. The results show that ad- versarial training is effective universally, and PQAT further improves the performance. 1 Introduction Neural networks have achieved superior perfor- mance on many tasks, but they are vulnerable to adversarial examples (Szegedy et al., 2014) – ex- amples that have been mixed with certain perturba- tions. Adversarial training (AT) (Goodfellow et al., 2015) uses both clean and adversarial examples to improve the robustness of the model for image classification. In the field of NLP, Miyato et al. (2017) have applied adversarial training on text classification tasks and improved the model performance. From then on, many AT methods has been proposed (Wu et al., 2017; Yasunaga et al., 2018; Bekoulis et al., 2018; Zhu et al., 2020; Jiang et al., 2019; Pereira et al., 2020; Liu et al., 2020). They mostly adopt a general AT strategy, but focus less on the adaptation of AT to NLP tasks. To explore this adaptation, in this work, we aim to apply adversarial training Passage: ... The rock cycle is an important concept in geology which illustrates the relationships between these three types of rock, and magma. When a rock crystallizes from melt (magma and/or lava), it is an igneous rock. ... Question: An igneous rock is a rock that crystallizes from what? Table 1: An example from the SQuAD dataset. We highlight two words rock and igneous for better demon- stration. The words with the same color are injected with the same perturbation by PQAT. The different oc- currences of the same word (for example, rock in pas- sage and question) are perturbed differently depending on their roles. on machine reading comprehension (MRC) tasks, which exhibit complex NLP characteristics. The objective of MRC is to let a machine read the given passages and ask it to answer the related questions. There are …

Content truncated to 500 words