Comparative Analysis of Neural QA models on SQuAD

Proceedings of the Workshop on Machine Reading for Question Answering, pages 89–97 Melbourne, Australia, July 19, 2018. c⃝2018 Association for Computational Linguistics 89 Comparative Analysis of Neural QA models on SQuAD Soumya Wadhwa Khyathi Raghavi Chandu Eric Nyberg Language Technologies Institute, Carnegie Mellon University {soumyaw, kchandu, en09}@andrew.cmu.edu Abstract The task of Question Answering has gained prominence in the past few decades for testing the ability of machines to un- derstand natural language. Large datasets for Machine Reading have led to the de- velopment of neural models that cater to deeper language understanding compared to information retrieval tasks. Different components in these neural architectures are intended to tackle different challenges. As a first step towards achieving gen- eralization across multiple domains, we attempt to understand and compare the peculiarities of existing end-to-end neu- ral models on the Stanford Question An- swering Dataset (SQuAD) by performing quantitative as well as qualitative analysis of the results attained by each of them. We observed that prediction errors reflect cer- tain model-specific biases, which we fur- ther discuss in this paper. 1 Introduction Machine Reading is a task in which a model reads a piece of text and attempts to formally represent it or performs a downstream task like Question An- swering (QA). Neural approaches to the latter have gained a lot of prominence especially owing to the recent spur in developing and publicly releasing large datasets on Machine Reading and Compre- hension (MRC). These datasets are created from different underlying sources such as web resources in MS MARCO (Nguyen et al., 2016); trivia and web in QUASAR-S and QUASAR-T (Dhin- gra et al., 2017), SearchQA (Dunn et al., 2017), TriviaQA (Joshi et al., 2017); news articles in CNN/Daily Mail (Chen et al.), NewsQA (Trischler et al., 2016) and stories in NarrativeQA (Koˇcisk`y et al., 2017). Another common source is large un- structured text documents from Wikipedia such as in SQuAD (Rajpurkar et al., 2016), WikiReading (Hewlett et al., 2016) and WikiHop (Welbl et al., 2017). These different sources implicitly affect the nature and properties of questions and answers in these datasets. Based on the dataset, certain neural models capitalize on these biases while others are unable to. The ability to generalize across differ- ent sources and domains is a desirable character- istic for any machine reading system. Evaluating and analyzing systems on QA tasks can lead to in- sights for advancements in machine reading and natural language understanding, and Pe˜nas et al. (2011) have also previously worked on this. One of the first large MRC datasets (over 100k QA pairs) is the Stanford Question Answering Dataset (SQuAD) (Rajpurkar et al., 2016). For its collection, different sets of crowd-workers for- mulated questions and answers using passages ob- tained from ∼500 Wikipedia articles. The answer to each question is a span in the given passage, and many effective neural QA models have been devel- oped for this dataset. Our main focus in this work is to perform comparative subjective and empirical analysis of errors in answer …

Content truncated to 500 words