Peek Across: Improving Multi-Document Modeling via Cross-Document Question-Answering

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 1970–1989 July 9-14, 2023 ©2023 Association for Computational Linguistics Peek Across: Improving Multi-Document Modeling via Cross-Document Question-Answering Avi Caciularu1∗ Matthew E. Peters2 Jacob Goldberger1 Ido Dagan1 Arman Cohan2,3 1Bar-Ilan University, Ramat-Gan, Israel 2Allen Institute for AI, Seattle, WA 3Yale University, New Haven, CT avi.c33@gmail.com, arman.cohan@yale.edu Abstract The integration of multi-document pre-training objectives into language models has resulted in remarkable improvements in multi-document downstream tasks. In this work, we propose extending this idea by pre-training a generic multi-document model from a novel cross- document question answering pre-training ob- jective. To that end, given a set (or cluster) of topically-related documents, we systemati- cally generate semantically-oriented questions from a salient sentence in one document and challenge the model, during pre-training, to answer these questions while “peeking" into other topically-related documents. In a sim- ilar manner, the model is also challenged to recover the sentence from which the ques- tion was generated, again while leveraging cross-document information. This novel multi- document QA formulation directs the model to better recover cross-text informational rela- tions, and introduces a natural augmentation that artificially increases the pre-training data. Further, unlike prior multi-document models that focus on either classification or summa- rization tasks, our pre-training objective formu- lation enables the model to perform tasks that involve both short text generation (e.g., QA) and long text generation (e.g., summarization). Following this scheme, we pre-train our model – termed QAMDEN – and evaluate its perfor- mance across several multi-document tasks, in- cluding multi-document QA, summarization, and query-focused summarization, yielding im- provements of up to 7%, and significantly out- performs zero-shot GPT-3.5 and GPT-4.1 1 Introduction Among recent NLP research, multi-document pro- cessing is gaining increasing attention, due to the need to handle and process an increasing amount of textual data and available documents online. A ∗ Work partly done as an intern at AI2. 1Our code is available at https://github.com/ aviclu/peekacross. (1) a set of related documents (2) context documents Pre-train by querying the context documents (3) held-out document Q: Where did Karol go to? A: She went to Sandra’s wedding. (4) (5) (6) split Figure 1: Illustration of our pre-training and data gen- eration. Per a considered set of related documents (1) which we split into context documents (2) and a held-out document (3), we select the most salient sentence (4) that is used for generating a question-answer pair (5). Then, we pre-train a model by generating the proper answer and the salient sentence, given the question and the context documents (6). number of prominent applications that are con- cerned with aggregating information from multiple texts are multi-document summarization (Fabbri et al., 2019; Zhao et al., 2020), query-focused multi- document summarization (Xu and Lapata, 2020; Pasunuru et al., 2021a), and multi-hop question answering (Yang et al., 2018; Welbl et al., 2018). These tasks remain challenging mostly since ex- isting NLP models are designed to handle single texts, rather than processing multiple documents …

Content truncated to 500 words