Adversarial Domain Adaptation for Machine Reading Comprehension

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 2510–2520, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 2510 Adversarial Domain Adaptation for Machine Reading Comprehension Huazheng Wang1∗, Zhe Gan2, Xiaodong Liu3, Jingjing Liu2, Jianfeng Gao3, Hongning Wang1 1University of Virginia, 2Microsoft Dynamics 365 AI Research, 3Microsoft Research {hw7ww,hw5x}@virginia.edu, {zhe.gan,xiaodl,jingjl,jfgao}@microsoft.com Abstract In this paper, we focus on unsupervised do- main adaptation for Machine Reading Com- prehension (MRC), where the source domain has a large amount of labeled data, while only unlabeled passages are available in the target domain. To this end, we propose an Adversarial Domain Adaptation framework (AdaMRC), where (i) pseudo questions are first generated for unlabeled passages in the target domain, and then (ii) a domain classifier is incorporated into an MRC model to predict which domain a given passage-question pair comes from. The classifier and the passage- question encoder are jointly trained using ad- versarial learning to enforce domain-invariant representation learning. Comprehensive eval- uations demonstrate that our approach (i) is generalizable to different MRC models and datasets, (ii) can be combined with pre-trained large-scale language models (such as ELMo and BERT), and (iii) can be extended to semi- supervised learning. 1 Introduction Recently, many neural network models have been developed for Machine Reading Comprehension (MRC), with performance comparable to human in specific settings (Gao et al., 2019). However, most state-of-the-art models (Seo et al., 2017; Liu et al., 2018; Yu et al., 2018) rely on large amount of human-annotated in-domain data to achieve the desired performance. Although there exists a number of large-scale MRC datasets (Rajpurkar et al., 2016; Trischler et al., 2016; Bajaj et al., 2016; Zhang et al., 2018), collecting such high- quality datasets is expensive and time-consuming, which hinders real-world applications for domain- specific MRC. ∗ Most of this work was done when the first author was an intern at Microsoft Dynamics 365 AI Research. Therefore, the ability to transfer an MRC model trained in a high-resource domain to other low- resource domains is critical for scalable MRC. While it is difficult to collect annotated question- answer pairs in a new domain, it is generally feasi- ble to obtain a large amount of unlabeled text in a given domain. In this work, we focus on adapting an MRC model trained in a source domain to other new domains, where only unlabeled passages are available. This domain adaptation issue has been a main challenge in MRC research, and the only exist- ing work that investigated this was the two-stage synthesis network (SynNet) proposed in Golub et al. (2017). Specifically, SynNet first generates pseudo question-answer pairs in the target domain, and then uses the generated data as augmenta- tion to fine-tune a pre-trained MRC model. How- ever, the source-domain labeled data and target- domain pseudo data are directly combined without considering domain differences (see Figure 1(a), where the two feature distributions in two domains are independently clustered). Directly transfer- …

Content truncated to 500 words