Template-Based Question Generation from Retrieved Sentences for Improved Unsupervised Question Answering

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4508–4513 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 4508 Template-Based Question Generation from Retrieved Sentences for Improved Unsupervised Question Answering Alexander R. Fabbri† 1,2 Patrick Ng‡1 Zhiguo Wang‡ Ramesh Nallapati‡ Bing Xiang‡ [†]Yale University [‡]AWS AI Labs alexander.fabbri@yale.edu, {patricng, zhiguow, rnallapa, bxiang}@amazon.com Abstract Question Answering (QA) is in increasing demand as the amount of information avail- able online and the desire for quick access to this content grows. A common approach to QA has been to fine-tune a pretrained lan- guage model on a task-specific labeled dataset. This paradigm, however, relies on scarce, and costly to obtain, large-scale human-labeled data. We propose an unsupervised approach to training QA models with generated pseudo- training data. We show that generating ques- tions for QA training by applying a simple template on a related, retrieved sentence rather than the original context sentence improves downstream QA performance by allowing the model to learn more complex context-question relationships. Training a QA model on this data gives a relative improvement over a pre- vious unsupervised model in F1 score on the SQuAD dataset by about 14%, and 20% when the answer is a named entity, achieving state- of-the-art performance on SQuAD for unsuper- vised QA. 1 Introduction Question Answering aims to answer a question based on a given knowledge source. Recent ad- vances have driven the performance of QA sys- tems to above or near-human performance on QA datasets such as SQuAD (Rajpurkar et al., 2016) and Natural Questions (Kwiatkowski et al., 2019) thanks to pretrained language models such as BERT (Devlin et al., 2019), XLNet (Yang et al., 2019) and RoBERTa (Liu et al., 2019). Fine-tuning these language models, however, requires large- scale data for fine-tuning. Creating a dataset for ev- ery new domain is extremely costly and practically infeasible. The ability to apply QA models on out- of-domain data in an efficient manner is thus very 1Equal contribution 2Work done during internship at the AWS AI Labs Figure 1: Question Generation Pipeline: the original context sentence containing a given answer is used as a query to retrieve a related sentence containing match- ing entities, which is input into our question-style con- verter to create QA training data. desirable. This problem may be approached with domain adaptation or transfer learning techniques (Chung et al., 2018) as well as data augmentation (Yang et al., 2017; Dhingra et al., 2018; Wang et al., 2018; Alberti et al., 2019). However, here we ex- pand upon the recently introduced task of unsuper- vised question answering (Lewis et al., 2019) to examine the extent to which synthetic training data alone can be used to train a QA model. In particular, we focus on the machine reading comprehension setting in which the context is a given paragraph, and the QA model can only access this paragraph to answer a question. Furthermore, we work on extractive QA, where the answer is as- sumed to be a …

Content truncated to 500 words