Long Context Question Answering via Supervised Contrastive Learning

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2872 - 2879 July 10-15, 2022 ©2022 Association for Computational Linguistics Long Context Question Answering via Supervised Contrastive Learning Avi Caciularu♣∗ Ido Dagan♣ Jacob Goldberger♠ Arman Cohan♦,♥ ♣Computer Science Department, Bar-Ilan University, Ramat-Gan, Israel ♠Faculty of Engineering, Bar-Ilan University, Ramat-Gan, Israel ♦Allen Institute for AI, Seattle, WA ♥Paul G. Allen School of Computer Science, University of Washington, Seattle, WA avi.c33@gmail.com, armanc@allenai.org dagan@cs.biu.ac.il, jacob.goldberger@biu.ac.il Abstract Long-context question answering (QA) tasks require reasoning over a long document or multiple documents. Addressing these tasks often benefits from identifying a set of evi- dence spans (e.g., sentences), which provide supporting evidence for answering the ques- tion. In this work, we propose a novel method for equipping long-context QA models with an additional sequence-level objective for better identification of the supporting evidence. We achieve this via an additional contrastive super- vision signal in finetuning, where the model is encouraged to explicitly discriminate support- ing evidence sentences from negative ones by maximizing question-evidence similarity. The proposed additional loss exhibits consistent improvements on three different strong long- context transformer models, across two chal- lenging question answering benchmarks – Hot- potQA and QAsper.1 1 Introduction Answering questions that require reasoning over a long sequence, such over long documents or multiple documents, is a challenging task (Pang et al., 2021). Research in this domain mostly in- cludes tasks that involve multiple text segments, over benchmarks like HotpotQA (Yang et al., 2018) and QAsper (Dasigi et al., 2021). HotpotQA is a multi-hop QA benchmark over multiple paragraphs from Wikipedia, while QAsper involves reading comprehension from long academic papers, where relevant information on a question could be spread across the paper. Given the task complexity (Choi et al., 2017), benchmarks often provide an additional set of evi- dence spans, such as sentences or paragraphs, for a given question answer pair. This breaks down the long-context QA task, adding a preliminary ∗ Work partly done as an intern at AI2. 1Our code is available at https://github.com/ aviclu/long-context-qa-contrast. evidence span detection, which is crucial for suc- cessfully finding the correct answer, and also poten- tially helps in model interpretability. In this work, we propose a method for improving long-context QA via leveraging such evidence spans, by maxi- mizing their similarity with the question. Since identifying the evidences provides rele- vant information for answering the question, prior work showed that jointly training models to per- form evidence span extraction in addition to answer generation is important for achieving high perfor- mance (Yang et al., 2018; Dasigi et al., 2021). To jointly perform evidence extraction and question answering, models utilize sentence representations (marker tokens) in the input; the final layer repre- sentation corresponding to these markers is then passed through a classification layer and is opti- mized using the cross-entropy loss in conjunction with the answer extraction/generation loss. We conjecture and demonstrate (see Table 1) that this objective does not sufficiently capture relationships between the …

Content truncated to 500 words