Harvesting and Refining Question-Answer Pairs for Unsupervised QA

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6719–6728 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 6719 Harvesting and Refining Question-Answer Pairs for Unsupervised QA Zhongli Li†∗, Wenhui Wang‡, Li Dong‡, Furu Wei‡, Ke Xu† †Beihang University ‡Microsoft Research {lizhongli@,kexu@nlsde.}buaa.edu.cn {wenwan,lidong1,fuwei}@microsoft.com Abstract Question Answering (QA) has shown great success thanks to the availability of large- scale datasets and the effectiveness of neu- ral models. Recent research works have at- tempted to extend these successes to the set- tings with few or no labeled data available. In this work, we introduce two approaches to improve unsupervised QA. First, we harvest lexically and syntactically divergent questions from Wikipedia to automatically construct a corpus of question-answer pairs (named as REFQA). Second, we take advantage of the QA model to extract more appropriate an- swers, which iteratively refines data over RE- FQA. We conduct experiments1 on SQuAD 1.1, and NewsQA by fine-tuning BERT with- out access to manually annotated data. Our ap- proach outperforms previous unsupervised ap- proaches by a large margin and is competitive with early supervised models. We also show the effectiveness of our approach in the few- shot learning setting. 1 Introduction Extractive question answering aims to extract a span from the given document to answer the ques- tion. Rapid progress has been made because of the release of large-scale annotated datasets (Ra- jpurkar et al., 2016, 2018; Joshi et al., 2017), and well-designed neural models (Wang and Jiang, 2016; Seo et al., 2016; Yu et al., 2018). Recently, unsupervised pre-training of language models on large corpora, such as BERT (Devlin et al., 2019), has brought further performance gains. However, the above approaches heavily rely on the availability of large-scale datasets. The collection of high-quality training data is time- consuming and requires significant resources, es- ∗Contribution during internship at Microsoft Research. 1The code and data are available at https://github. com/Neutralzz/RefQA. pecially for new domains or languages. In order to tackle the setting in which no training data avail- able, Lewis et al. (2019) leverage unsupervised machine translation to generate synthetic context- question-answer triples. The paragraphs are sam- pled from Wikipedia. NER and noun chunkers are employed to identify answer candidates. Cloze questions are first extracted from the sentences of the paragraph, and then translated into natural questions. However, there are a lot of lexical over- laps between the generated questions and the para- graph. Similar lexical and syntactic structures ren- der the QA model tend to predict the answer just by word matching. Moreover, the answer cate- gory is limited to the named entity or noun phrase, which restricts the coverage of the learnt model. In this work, we present two approaches to im- prove the quality of synthetic context-question- answer triples. First, we introduce the REFQA dataset, which harvests lexically and syntactically divergent questions from Wikipedia by using the cited documents. As shown in Figure 1, the sentence (statement) in Wikipedia and its cited documents are semantically consistent, but writ- ten with …

Content truncated to 500 words