DiffQG: Generating Questions to Summarize Factual Changes

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 3088–3101 May 2-6, 2023 ©2023 Association for Computational Linguistics DIFFQG: Generating Questions to Summarize Factual Changes Jeremy R. Cole1∗ Palak Jain1∗ Julian Martin Eisenschlos1 Michael J.Q. Zhang3 Eunsol Choi3 Bhuwan Dhingra1,2 1 Google Research 2 Duke University 3 The University of Texas at Austin {jrcole,palakj,eisenjulian,bdhingra}@google.com {mjqzhang,eunsol}@utexas.edu Abstract Identifying the difference between two ver- sions of the same article is useful to update knowledge bases and to understand how arti- cles evolve. Paired texts occur naturally in di- verse situations: reporters write similar news stories and maintainers of authoritative web- sites must keep their information up to date. We propose representing factual changes be- tween paired documents as question-answer pairs, where the answer to the same question differs between two versions. We find that question-answer pairs can flexibly and con- cisely capture the updated contents. Provided with paired documents, annotators identify questions that are answered by one passage but answered differently or cannot be answered by the other. We release DIFFQG which con- sists of 759 QA pairs and 1153 examples of paired passages with no factual change. These questions are intended to be both unambigu- ous and information-seeking and involve com- plex edits, pushing beyond the capabilities of current question generation and factual change detection systems. Our dataset summarizes the changes between two versions of the docu- ment as questions and answers, studying auto- matic update summarization in a novel way. 1 Introduction Given a pair of statements, how can we iden- tify the difference in their information content? This problem has existed in different forms across NLP research, such as recognizing textual entail- ment (Dagan et al., 2010) and natural language inference (Bowman et al., 2015). The initial fo- cus of this type of research was finding the logical implication relations between sentences. More recently, specialized entailment-like re- sources and models have been applied to fact veri- fication (Thorne et al., 2018b) with applications to science, education and journalism. This trend has ∗Equal Contribution. Figure 1: DIFFQG consists of paired Wikipedia pas- sages that correspond to factual edits. The goal is to generate a discriminating question given an answer span such that the question is answerable by one of the passages but not the other or yields different answers. exposed the limited transfer between logical entail- ment and general factual change detection (Thorne et al., 2018a) as well as the need for interpretable models for this task (Kumar and Talukdar, 2020). Wikipedia revisions across time provide a large scale and highly available source of sentence pairs, leading to new resources such as WIKIATOMICED- ITS (Faruqui et al., 2018) and VITAMINC (Schus- ter et al., 2021). However, prior work is limited to minimal changes that concern only a single fac- tual addition or change. We introduce DIFFQG, a manually annotated dataset spanning changes over multiple years. DIFFQG consists of paired passages with complex factual changes including multiple additions and deletions within the same example. …

Content truncated to 500 words