Unsupervised Adaptation of Question Answering Systems via Generative Self-training

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 1148–1157, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 1148 Unsupervised Adaptation of Question Answering Systems via Generative Self-training Steven J. Rennie, Etienne Marcheret, Neil Mallinar, David Nahamoo, and Vaibhava Goel Pryon Inc., Brooklyn, New York, New York. {srennie, emarcheret, nmallinar, dnahamoo, vgoel}@pryon.com Abstract BERT-era question answering systems have recently achieved impressive performance on several question-answering (QA) tasks. These systems are based on representations that have been pre-trained on self-supervised tasks such as word masking and sentence entailment, us- ing massive amounts of data. Nevertheless, additional pre-training closer to the end-task, such as training on synthetic QA pairs, has been shown to improve performance. While re- cent work has considered augmenting labelled data and leveraging large unlabelled datasets to generate synthetic QA data, directly adapt- ing to target data has received little attention. In this paper we investigate the iterative gener- ation of synthetic QA pairs as a way to realize unsupervised self adaptation. Motivated by the success of the roundtrip consistency method for filtering generated QA pairs, we present it- erative generalizations of the approach, which maximize an approximation of a lower bound on the probability of the adaptation data. By adapting on synthetic QA pairs generated on the target data, our method is able to im- prove QA systems significantly, using an or- der of magnitude less synthetic data and train- ing computation than existing augmentation approaches. 1 Introduction Supervised self-training methods have transformed applied machine learning recently. Such tasks serve as “pre-training” for related downstream tasks, and have proven to be essential to attaining state-of-the- art performance, particularly in NLP. BERT-era Transformer-based question answer- ing systems have recently achieved impressive performance on several question-answering (QA) tasks. These systems are based on representations that have been pre-trained on self-supervised tasks such as word masking and sentence entailment, us- ing massive amounts of data (Devlin et al., 2018; Liu et al., 2019; Yang et al., 2019; Dong et al., 2019; Radford et al., 2019). Nevertheless, additional pre- training closer to the end-task, such as training on synthetic QA pairs, has been shown to improve per- formance (Alberti et al., 2019; Dong et al., 2019). While recent work has considered augmenting la- belled data and leveraging large unlabelled datasets to generate synthetic QA data, directly adapting to target data has, to our knowledge, not been inves- tigated in the context of BERT-era modeling and performance levels. Recently, roundtrip consistency (RTC) was intro- duced as a criteria for filtering synthetic question- answer pairs on unlabelled data, and has demon- strated solid gains when applied to large unlabelled datasets to generate millions of RTC-validated pairs as task-specific pre-training data (Alberti et al., 2019). Such unsupervised self-training can be an effective way to de-emphasize low confidence pre- dictions, adapt to the target input distribution, dis- till decoding procedures, and instill input response invariances. In this paper we present new theoretical justifi- cation for RTC, and explore novel iterative gener- alizations …

Content truncated to 500 words