Dynamic Sampling Strategies for Multi-Task Reading Comprehension

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 920–924 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 920 Dynamic Sampling Strategies for Multi-Task Reading Comprehension Ananth Gottumukkala† Dheeru Dua† Sameer Singh† Matt Gardner‡ †University of California, Irvine, USA ‡Allen Institute for Artificial Intelligence, Irvine, California, USA {agottumu, ddua, sameer}@uci.edu mattg@allenai.edu Abstract Building general reading comprehension sys- tems, capable of solving multiple datasets at the same time, is a recent aspirational goal in the research community. Prior work has focused on model architectures or generaliza- tion to held out datasets, and largely passed over the particulars of the multi-task learning set up. We show that a simple dynamic sam- pling strategy, selecting instances for training proportional to the multi-task model’s current performance on a dataset relative to its single- task performance, gives substantive gains over prior multi-task sampling strategies, mitigat- ing the catastrophic forgetting that is common in multi-task learning. We also demonstrate that allowing instances of different tasks to be interleaved as much as possible between each epoch and batch has a clear benefit in multi- task performance over forcing task homogene- ity at the epoch or batch level. Our final model shows greatly increased performance over the best model on ORB, a recently-released multi- task reading comprehension benchmark. 1 Introduction Building multi-task reading comprehension sys- tems has received significant attention and been a focus of active research (Talmor and Berant, 2019; Xu et al., 2019). These approaches mostly focus on model architecture improvements or generalizabil- ity to new tasks or domains. While these contribu- tions are important, it is also important to explore the optimal way to structure training; as we will show, training on instances from diverse datasets (tasks) means that unlike in a single-task setting, ample instances from each task distribution must be represented during training to properly capture that diversity. We explore 2 fundamental aspects of structuring multi-task training: how many in- stances are sampled from each task per epoch and how those instances are organized within the epoch. We investigate the importance of this structuring by training a multi-task model on the 8 datasets from ORB (Dua et al., 2019b), a recent multi-task reading comprehension benchmark. We first explore the sampling distribution over datasets at each epoch: how many instances from each dataset should be used to train. Prior work has typically either made this a uniform distri- bution over datasets (implicitly favoring smaller datasets), a distribution proportional to the sizes of the datasets (implicitly favoring larger datasets), or some combination of the two. Because these sampling strategies favor some datasets over oth- ers, they can lead to catastrophic forgetting in the non-favored datasets. We introduce a dynamic sam- pling strategy that selects instances from a dataset with probability proportional to the gap between its current performance on some metric (like EM or F1 score) and measured single-task performance of the same model on that dataset. By adjusting the sampling distribution over the course of training ac- cording …

Content truncated to 500 words