SQuAD2-CR: Semi-supervised Annotation for Cause and Rationales for Unanswerability in SQuAD 2.0

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 5425–5432 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 5425 SQuAD2-CR: Semi-supervised Annotation for Cause and Rationales for Unanswerability in SQuAD 2.0 Gyeongbok Lee1, Seung-won Hwang2, Hyunsouk Cho1 1Knowledge AI Lab., NCSOFT Co., 2Yonsei University gbpalace@ncsoft.com, seungwonh@yonsei.ac.kr, dakgalbi@ncsoft.com Abstract Existing machine reading comprehension models are reported to be brittle for adversarially perturbed questions when optimizing only for accuracy, which led to the creation of new reading comprehension benchmarks, such as SQuAD 2.0 which contains such type of questions. However, despite the super-human accuracy of existing models on such datasets, it is still unclear how the model predicts the answerability of the question, potentially due to the absence of a shared annotation for the explanation. To address such absence, we release SQuAD2-CR dataset, which contains annotations on unanswerable questions from the SQuAD 2.0 dataset, to enable an explanatory analysis of the model prediction. Specifically, we annotate (1) explanation on why the most plausible answer span cannot be the answer and (2) which part of the question causes unanswerability. We share intuitions and experimen- tal results that how this dataset can be used to analyze and improve the interpretability of existing reading comprehension model behavior. Keywords: SQuAD 2.0, Machine Reading Comprehension, Corpus Annotation, Model Interpretability, Evaluation 1. Introduction The machine reading comprehension (MRC) task aims to find useful information from unstructured text queried in the form of natural language. To solve this task, a model needs the ability to find the context associated with the question and infer the correct answer. Recently, data-driven learning methods are being actively studied, as many large- scale benchmark data are released and various resources on the web can be easily accessed and utilized. Among MRC benchmarks, Stanford Question Answering Dataset (SQuAD) is the most widely adopted for evalu- ating the reading comprehension capabilities of a model, which evaluates how well the model predicts the answer span for a paragraph, given a natural language problem. As it is a large-scale, high-quality set of annotations ob- tained from crowdsourcing, many state-of-the-art methods use this dataset to train their models and show their effec- tiveness compared to previous approaches. However, the first version of SQuAD was designed to have an answer span for all problems, which trained models to find the most relevant span regardless of whether the cor- rect answer was actually inferred from the question. This bias is reported to degrade the robustness of the model for adversarial perturbed questions or paragraph. To solve this problem, SQuAD 2.0 was released, to include unanswer- able problems obtained by crowdsourcing human perturba- tions, such as changing the word in a question or adding a question that is not related to the problem. Although pretrained contextualized embedding, obtained through language modeling from large corpora, has en- abled superhuman performance for both SQuAD 1.0 and 2.0, their robustness with respect to model behavior has been understudied. To illustrate, Figure 1 shows an ex- …

Content truncated to 500 words