MultiQA: An Empirical Investigation of Generalization and Transfer in Reading Comprehension

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4911–4921 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 4911 MULTIQA: An Empirical Investigation of Generalization and Transfer in Reading Comprehension Alon Talmor1,2 Jonathan Berant1,2 1School of Computer Science, Tel-Aviv University 2Allen Institute for Artificial Intelligence {alontalmor@mail,joberant@cs}.tau.ac.il Abstract A large number of reading comprehension (RC) datasets has been created recently, but little analysis has been done on whether they generalize to one another, and the extent to which existing datasets can be leveraged for improving performance on new ones. In this paper, we conduct such an investigation over ten RC datasets, training on one or more source RC datasets, and evaluating generaliza- tion, as well as transfer to a target RC dataset. We analyze the factors that contribute to gen- eralization, and show that training on a source RC dataset and transferring to a target dataset substantially improves performance, even in the presence of powerful contextual repre- sentations from BERT (Devlin et al., 2019). We also find that training on multiple source RC datasets leads to robust generalization and transfer, and can reduce the cost of example collection for a new RC dataset. Following our analysis, we propose MULTIQA, a BERT- based model, trained on multiple RC datasets, which leads to state-of-the-art performance on five RC datasets. We share our infrastructure for the benefit of the research community. 1 Introduction Reading comprehension (RC) is concerned with reading a piece of text and answering questions about it (Richardson et al., 2013; Berant et al., 2014; Hermann et al., 2015; Rajpurkar et al., 2016). Its appeal stems both from the clear ap- plication it proposes, but also from the fact that it allows to probe many aspects of language un- derstanding, simply by posing questions on a text document. Indeed, this has led to the creation of a large number of RC datasets in recent years. While each RC dataset has a different focus, there is still substantial overlap in the abilities re- quired to answer questions across these datasets. Nevertheless, there has been relatively little work (Min et al., 2017; Chung et al., 2018; Sun et al., 2018) that explores the relations between the dif- ferent datasets, including whether a model trained on one dataset generalizes to another. This re- search gap is highlighted by the increasing interest in developing and evaluating the generalization of language understanding models to new setups (Yo- gatama et al., 2019; Liu et al., 2019). In this work, we conduct a thorough empiri- cal analysis of generalization and transfer across 10 RC benchmarks. We train models on one or more source RC datasets, and then evaluate their performance on a target test set, either without any additional target training examples (general- ization) or with additional target examples (trans- fer). We experiment with DOCQA (Clark and Gardner, 2018), a standard and popular RC model, as well as a model based on BERT (Devlin et al., 2019), which provides powerful …

Content truncated to 500 words