Systematic Error Analysis of the Stanford Question Answering Dataset

Proceedings of the Workshop on Machine Reading for Question Answering, pages 12–20 Melbourne, Australia, July 19, 2018. c⃝2018 Association for Computational Linguistics 12 Systematic Error Analysis of the Stanford Question Answering Dataset Marc-Antoine Rondeau Microsoft Research Montr´eal, Qu´ebec, Canada marondea@microsoft.com Timothy J. Hazen Microsoft Research Cambridge, Massachusetts, USA tj.hazen@microsoft.com Abstract We analyzed the outputs of multiple ques- tion answering (QA) models applied to the Stanford Question Answering Dataset (SQuAD) to identify the core challenges for QA systems on this data set. Through an iterative process, challenging aspects were hypothesized through qualitative analysis of the common error cases. classifier was then constructed to predictA whether SQuAD test examples were likely to be difficult for systems to answer based on features associated with the hypothe- sized aspects. The classifier’s performance was used to accept or reject each aspect as an indicator of difficulty. With this ap- proach, we ensured that our hypotheses were systematically tested and not simply accepted based on our pre-existing biases. Our explanations are not accepted based on human evaluation of individual exam- ples. This process also enabled us to iden- tify the primary QA strategy learned by the models, i.e., systems determined the ac- ceptable answer type for a question and then selected the acceptable answer span of that type containing the highest density of words present in the question within its local vicinity in the passage. 1 Introduction Since the introduction of the Stanford Question Answering Dataset (SQuAD, Rajpurkar et al., 2016), research groups have directed significant efforts towards achieving a high position on the SQuAD leaderboard.1 This competition has re- 1https://rajpurkar.github.io/ SQuAD-explorer/ sulted in many new models for question answering using machine reading comprehension. Within SQuAD, a single test example consists of three components: a question, a text passage and an answer. The answer is a span extracted from the passage answering the question. Ques- tions were created by human annotators, who were shown a passage and asked to produce question and answer pairs. In performing the question an- swering task, the best performing systems em- ployed complex attention flow mechanisms for matching questions to substrings of the text pas- sage. These models, while varied, all belong to the same general family of neural network archi- tectures. In this work, we conducted a systematic error analysis on the development set of SQuAD to ex- plain the common failures and successes of some of these models; the results can be expected to generalize to the entire family. Our goal was to ex- plain the models’ failures and successes using well defined features automatically extracted from ex- amples. We wanted to use simple features, such as word identity, over complex features. We wanted to avoid explanations based on the human strat- egy used to answer a question, or complex features that cannot be extracted automatically, such as rea- soning, common sense or external knowledge. Fi- nally, we wanted to isolate a passages’ readability from the strategy required to answer questions. Our methodology used classifiers to predict the difficulty …

Content truncated to 500 words