Back-Training excels Self-Training at Unsupervised Domain Adaptation of Question Generation and Passage Retrieval

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7064–7078 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 7064 Back-Training excels Self-Training at Unsupervised Domain Adaptation of Question Generation and Passage Retrieval Devang Kulshreshtha1,2, Robert Belfer2, Iulian Vlad Serban2, Siva Reddy1,3 1Mila/McGill University 2Korbit Technologies 3Facebook CIFAR AI Chair {devang.kulshreshtha, siva.reddy}@mila.quebec, {iulian, robert}@korbit.ai Abstract In this work, we introduce back-training, an al- ternative to self-training for unsupervised do- main adaptation (UDA) from source to target domain. While self-training generates syn- thetic training data where natural inputs are aligned with noisy outputs, back-training re- sults in natural outputs aligned with noisy in- puts. This significantly reduces the gap be- tween the target domain and synthetic data distribution, and reduces model overfitting to the source domain. We run UDA experiments on question generation and passage retrieval from the Natural Questions domain to ma- chine learning and biomedical domains. We find that back-training vastly outperforms self- training by a mean improvement of 7.8 BLEU- 4 points on generation, and 17.6% top-20 re- trieval accuracy across both domains. We further propose consistency filters to remove low-quality synthetic data before training. We also release a new domain-adaptation dataset- MLQuestions containing 35K unaligned ques- tions, 50K unaligned passages, and 3K aligned question-passage pairs. 1 Introduction In domains such as education and medicine, collect- ing labeled data for tasks like question answering and generation requires domain experts, thereby making it expensive to build supervised models. Transfer learning can circumvent this limitation by exploiting models trained on other domains where labeled data is readily available (Bengio, 2012; Ruder et al., 2019). However, using these pre-trained models directly without adapting to the target domain often leads to poor generalization due to distributional shift (Zhao et al., 2019). To address this issue, these models are further trained on cheap synthetically generated labeled data by exploiting unlabeled data from target domain (Ram- poni and Plank, 2020). One such popular data aug- mentation method for unsupervised domain adap- tation (UDA) is self-training (Yarowsky, 1995). Synthetic Training Data Algorithm Input Output Question Generation (QG) Self-Training pu ∼ PT (p) ˆq ∼ PS(q|pu) Back-Training ˆp ∼ PS(p|qu) qu ∼ PT (q) Passage Retrieval (IR) Self-Training qu ∼ PT (q) ˆp ∼ PS(p|qu) Back-Training ˆq ∼ PS(q|pu) pu ∼ PT (p) Table 1: Self-Training and Back-Training for unsuper- vised domain adaptation of question generation and passage retrieval. In self-training, inputs are sampled from the target domain data distribution PT and their corresponding outputs are generated using a supervised model PS trained on the source domain. In back- training, the inverse happens: outputs are sampled from PT and their corresponding inputs are generated using PS. Notation: q and p denote questions and passages respectively, .u denotes samples from the target domain and ˆ. denotes the samples generated by a supervised model trained on the source domain. In self-training, given a pre-trained model that can perform the task of interest in a source domain and unlabeled data from the target domain, the pre- …

Content truncated to 500 words