Learning with Instance Bundles for Reading Comprehension

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7347–7357 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 7347 Learning with Instance Bundles for Reading Comprehension Dheeru Dua♥ Pradeep Dasigi♠ Sameer Singh♥ Matt Gardner♠ ♥University of California, Irvine, USA ♠Allen Institute for Artificial Intelligence ddua@uci.edu Abstract When training most modern reading compre- hension models, all the questions associated with a context are treated as being indepen- dent from each other. However, closely re- lated questions and their corresponding an- swers are not independent, and leveraging these relationships could provide a strong su- pervision signal to a model. Drawing on ideas from contrastive estimation, we intro- duce several new supervision losses that com- pare question-answer scores across multiple related instances. Specifically, we normalize these scores across various neighborhoods of closely contrasting questions and/or answers, adding a cross entropy loss term in addi- tion to traditional maximum likelihood esti- mation. Our techniques require bundles of related question-answer pairs, which we ei- ther mine from within existing data or cre- ate using automated heuristics. We empiri- cally demonstrate the effectiveness of train- ing with instance bundles on two datasets— HotpotQA and ROPES—showing up to 9% ab- solute gains in accuracy. 1 Introduction Machine learning models are typically trained with the assumption that the training instances sam- pled from some data distribution are independent and identically distributed. However, this assump- tion can cause the learner to ignore distinguish- ing cues (Dietterich et al., 1997) between related or minimally different questions associated with a given context, resulting in inconsistent model pre- dictions (Jia and Liang, 2017; Ribeiro et al., 2019; Asai and Hajishirzi, 2020). In cases like ROPES, where the dataset contains only minimally different questions, we see that the performance of a com- petitive model (RoBERTA) is close to random (Lin et al., 2019). One potential reason for this poor per- formance is that the model considers each question independently, instead of looking at differences be- tween related questions. Context: Marsilea is a genus of approximately 65 species of aquatic ferns of the family Marsileaceae. The name honours Italian naturalist Luigi Ferdinando Marsili (1656-1730) .... Brabejum is a genus of a single species of large evergreen tree, Brabejum stellatifolium in the Proteaceae, commonly called wild almond, bitter almond or ghoeboontjie. Question, q: Is the Marsilea or the Brabejum the genus of more individual species of plants? Answer, a: Marsilea Question, ˜q: Is the Marsilea or the Brabejum the genus of less individual species of plants? Answer, ˜a: Brabejum (a) Instance bundle created from HotpotQA. The QA pair, (˜q, ˜a) is heuristically generated from (q, a) in the dataset. 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 p(a|q) p(a|q) + p(a|q) MLE MLE (Augmented) Unlikelihood CE (b) Distribution over probability scores of gold QA pair, (a, q) normalized over the two questions {q, ˜q} in each instance bundle, in HotpotQA’s comparison type dev set. The higher value indi- cates that correct QA pair has a higher likelihood than the wrong …

Content truncated to 500 words