What’s the Issue Here?: Task-based Evaluation of Reader Comment Summarization Systems

What’s the Issue Here?: Task-based Evaluation of Reader Comment Summarization Systems Emma Barker, Monica Paramita, Adam Funk, Emina Kurtic, Ahmet Aker, Jonathan Foster, Mark Hepple and Robert Gaizauskas Department of Computer Science University of Sheffield, UK {initial.surname}@sheffield.ac.uk Abstract Automatic summarization of reader comments in on-line news is an extremely challenging task and a capability for which there is a clear need. Work to date has focussed on producing extractive summaries using well-known techniques imported from other areas of language processing. But are extractive summaries of comments what users really want? Do they support users in performing the sorts of tasks they are likely to want to perform with reader comments? In this paper we address these questions by doing three things. First, we offer a specification of one possible summary type for reader comment, based on an analysis of reader comment in terms of issues and viewpoints. Second, we define a task-based evaluation framework for reader comment summarization that allows summarization systems to be assessed in terms of how well they support users in a time-limited task of identifying issues and characterising opinion on issues in comments. Third, we describe a pilot evaluation in which we used the task-based evaluation framework to evaluate a prototype reader comment clustering and summarization system, demonstrating the viability of the evaluation framework and illustrating the sorts of insight such an evaluation affords. Keywords: Reader comment summarization, task-based evaluation, social media argumentation 1. Introduction A common feature of on-line news sites is reader comment – a facility whereby readers of a news story can engage in conversation with each other, discussing aspects of or reac- tions to a news story. Often, news stories attract hundreds or even thousands of reader comments within a relatively short period of time. The problem then arises of how to make sense of this sprawling, multi-threaded conversation. Intuitively, what one wants is some kind of summary or overview of the conversation, with the option of “drilling down” for more details. Since generating such an overview manually for every news story is clearly infeasible, auto- matic summarization offers promise here. And indeed, sev- eral authors have already proposed systems for summariz- ing reader comments (e.g. (Khabiri et al., 2011; Ma et al., 2012; Llewellyn et al., 2014)). Their systems are broadly similar, first topically clustering comments, then ranking comments within clusters and finally selecting top-ranked comments to form an extractive summary. This extraction-oriented work begins with the assumption that topically grouped and ranked comments or extractive summaries are to likely to be useful to an end user. But it is true? To our knowledge there has been no attempt to investigate what an end user might want in a summary of reader comments. Furthermore, the evaluations proposed so far, despite in several cases being called user studies, are not task-based evaluations that might let us understand how well systems are meeting user needs. A different, but promising, line of work, not yet deployed in summarization systems, is that on argument mining. …

Content truncated to 500 words