Extending Multi-Document Summarization Evaluation to the Interactive Setting

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 657–677 June 6–11, 2021. ©2021 Association for Computational Linguistics 657 Extending Multi-Document Summarization Evaluation to the Interactive Setting Ori Shapira1, Ramakanth Pasunuru2, Hadar Ronen3, Mohit Bansal2, Yael Amsterdamer1, and Ido Dagan1 1Bar-Ilan University 2UNC Chapel Hill 3Peres Academic Center {obspp18, hadarg}@gmail.com {ram, mbansal}@cs.unc.edu {amstery, dagan}@cs.biu.ac.il Abstract Allowing users to interact with multi- document summarizers is a promising direction towards improving and customizing summary results. Different ideas for interac- tive summarization have been proposed in previous work but these solutions are highly divergent and incomparable. In this paper, we develop an end-to-end evaluation framework for interactive summarization, focusing on expansion-based interaction, which considers the accumulating information along a user session. Our framework includes a procedure of collecting real user sessions, as well as evaluation measures relying on summarization standards, but adapted to reflect interaction. All of our solutions and resources are available publicly as a benchmark, allowing compar- ison of future developments in interactive summarization, and spurring progress in its methodological evaluation. We demonstrate the use of our framework by evaluating and comparing baseline implementations that we developed for this purpose, which will serve as part of our benchmark. Our extensive experimentation and analysis motivate the proposed evaluation framework design and support its viability. 1 Introduction Large bodies of texts on a topic oftentimes con- tain extensive information that is challenging for a potential reader to handle. Traditionally, informa- tion seeking tasks, like search, question-answering (QA) and multi-document summarization (MDS), are single-round input-output processes that can serve the information seeker only to a limited ex- tent. This calls for an interactive setting where a user can guide the information gathering process. For search and QA, this type of research has been gaining momentum recently in areas such as ex- ploratory search (Marchionini, 2006) and conversa- tional QA (Reddy et al., 2019). For MDS, where interaction would allow a user to affect summary content, only sporadic works have been seen over the years (e.g., Leuski et al., 2003; Lin et al., 2010; Yan et al., 2011; Baumel et al., 2014; Christensen et al., 2014; Shapira et al., 2017; Handler and O’Connor, 2017). A key gap in the development and adoption of interactive sum- marization (denoted here INTSUMM) solutions is the lack of evaluation methodologies and bench- marks for meaningful comparison of systems, sim- ilarly to those for static (non-interactive) summa- rization (e.g., NIST, 2014). The previous works on interactive or customizable summarization of multi-document sets are distinct, with proprietary evaluations that do not admit comparison. Further- more, the evaluation processes are often not scal- able and replicable, or do not give a comprehensive enough assessment. In this paper we develop an end-to-end evalua- tion framework for INTSUMM systems. The frame- work starts with real user session collection on a system, via a concrete process of controlled crowd- sourcing that we designed for this task. The ses- sions are then measured to produce …

Content truncated to 500 words