Handling Anomalies of Synthetic Questions in Unsupervised Question Answering

Proceedings of the 28th International Conference on Computational Linguistics, pages 3441–3448 Barcelona, Spain (Online), December 8-13, 2020 3441 Handling Anomalies of Synthetic Questions in Unsupervised Question Answering Giwon Hong∗ Junmo Kang∗ Doyeon Lim∗ Sung-Hyon Myaeng School of Computing, KAIST Daejeon, Republic of Korea {gch02518, junmo.kang, dylim, myaeng}@kaist.ac.kr Abstract Advances in Question Answering (QA) research require additional datasets for new domains, languages, and types of questions, as well as for performance increases. Human creation of a QA dataset like SQuAD, however, is expensive. As an alternative, an unsupervised QA approach has been proposed so that QA training data can be generated automatically. However, the perfor- mance of unsupervised QA is much lower than that of supervised QA models. We identify two anomalies in the automatically generated questions and propose how they can be mitigated. We show our approach helps improve unsupervised QA significantly across a number of QA tasks. 1 Introduction Machine Reading Question Answering (MRQA) is a popular NLP task that attempts to answer ques- tions from associated texts. A popular challenge is SQuAD (Rajpurkar et al., 2016), which has 100K ⟨context, answer, question⟩ triples created by humans, and other challenging tasks and datasets have been created. However, it is costly to create a dataset for a new QA task or expand an existing one. Context Generated Question ... Montecarlo Automobile began manufacturing its first street legal GT car in 1989 ... How much does manufacturing its street legal GT car in 1989 ... ? ... it is hot in the summer and freezing in the winter ... When is it hot in the summer and freezing? Table 1: A copy-type question (green) and an unanswerable question (red). As an attempt to relieve the burden of human efforts, Lewis et al. (2019) proposed an unsupervised question answering (UQA) where the answers and questions are generated in an unsupervised way. The core of the approach is to create a cloze question from a context and then apply a translator to convert it into a natural question. This pioneering work provides a reasonable baseline for UQA but leaves a large room for improvement on question generation. We have identified two anomalies of the unsupervised question generation (UQG) model, in addition to the question-type issues (Kang et al., 2019; Lim et al., 2020). First, a large number of the tokens in the generated questions are copied from the answer-containing sentences in the contexts, producing copy-type questions. As a result, a QA model trained with a large number of these “easy” questions is prone to fail when confronted with questions not directly derivable from the contexts. Second, the UQG model sometimes creates unanswerable questions that are semantically too far away from the context, which only acts as noise when training a QA model. An example of the two anomaly types is shown in Table 1. The main thrust of this paper is to demonstrate that the anomalies of UQG can be overcome by means of relatively simple techniques, which produce higher quality questions that …

Content truncated to 500 words