Single-dataset Experts for Multi-dataset Question Answering

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6128–6137 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 6128 Single-dataset Experts for Multi-dataset Question Answering Dan Friedman Ben Dodge Danqi Chen Department of Computer Science, Princeton, NJ {dfriedman,bd4,danqic}@cs.princeton.edu Abstract Many datasets have been created for train- ing reading comprehension models, and a natural question is whether we can combine them to build models that (1) perform better on all of the training datasets and (2) gen- eralize and transfer better to new datasets. Prior work has addressed this goal by train- ing one network simultaneously on multiple datasets, which works well on average but is prone to over- or under-fitting different sub- distributions and might transfer worse com- pared to source models with more overlap with the target dataset. Our approach is to model multi-dataset question answering with an ensemble of single-dataset experts, by train- ing a collection of lightweight, dataset-specific adapter modules (Houlsby et al., 2019) that share an underlying Transformer model. We find that these Multi-Adapter Dataset Experts (MADE) outperform all our baselines in terms of in-distribution accuracy, and simple meth- ods based on parameter-averaging lead to bet- ter zero-shot generalization and few-shot trans- fer performance, offering a strong and ver- satile starting point for building new reading comprehension systems.1 1 Introduction The goal of reading comprehension is to create computer programs that can answer questions based on a single passage of text. Many reading comprehension datasets have been introduced over the years, and prior work has explored ways of training one network on multiple datasets to get a model that generalizes better to new distribu- tions (Talmor and Berant, 2019; Fisch et al., 2019; Khashabi et al., 2020). Our goal is to build a multi- dataset model that performs well on the training distributions and can also serve as a strong starting 1Our code and models are available at https:// github.com/princeton-nlp/MADE. θL θ1 ϕ1 i ϕ1 j ϕ1 k qi qj qk ai aj ak ϕL−1 i ϕL−1 j ϕL−1 k ψi ψj ψk … θ′ L θ′ 1 ϕ′ 1 qtgt ϕ′ L−1 ψ′ … atgt Figure 1: MADE consists of a set of dataset-specific adapters and classifiers {(φt, ψt)}|S| t=1 with a shared Transformer model θ. They are optimized jointly on a set of training datasets (left). To transfer to a new dataset (right), we either average the parameters of the adapters, or fine-tune the ensemble on the target dataset and take the weighted average at the end (Section 3). point for transfer learning to new datasets. Multi- dataset training provides a way to model the reg- ularities between datasets but it has the follow- ing shortcomings. First, multi-task models are liable to over- or under-fit different tasks (Got- tumukkala et al., 2020), which can result in worse in-distribution accuracy. Second, given a particular target dataset, multi-dataset models might achieve worse transfer performance compared to a special- ized model trained on a more similar source dataset. Our idea is to combine …

Content truncated to 500 words