A Comparison Study of Human Evaluated Automated Highlighting Systems

A Comparison Study of Human-Evaluated Automated Highlighting Systems Sasha Spala1, Franck Dernoncourt2, Walter Chang2, Carl Dockhorn1 1Adobe Systems, 2Adobe Research 345 Park Ave San Jose, CA 95110-2704 {sspala,dernonco,wachang,cdockhor}@adobe.com Abstract Automatic text highlighting aims to identify key portions that are most important to a reader. In this paper, we explore the use of ex- isting extractive summarization models for au- tomatically generating highlights; automatic highlight generation has not previously been addressed from this perspective. Evaluation studies typically rely on automated evaluation metrics as they are cheap to compute and scale well. However, these metrics are not designed to assess automated highlighting. We there- fore focus on human evaluations in this work. Our comparison of multiple summarization models used for automated highlighting ac- companied by human evaluation provides an approximate upper bound of the quality of fu- ture highlighting models. 1 Introduction Automatic text highlighting aims to identify key por- tions that are most important to a first-time reader. Our motivation for pursuing highlighting analysis stems from an apparent lack of current computa- tional research and machine learning applications dedicated to this type of reading aid. For clarity, we define document highlights as a brightly-colored overlay placed on top of a span of text in order to attract readers’ attention to the content of that text. We do not define any one specific color those high- lights must be, but our practice, as outlined in the methodology, makes use of yellow, green, and red highlights. Highlighting, when used correctly, can be an im- portant part of the reading process, and, in many cases, can aid a reader in information retention (Fowler and Barker, 1974). Because of its similarity to extractive summarization tasks, where highlights reflect the sentences retrieved for an extractive sum- mary, it follows that this is a relevant repurpose of these summarization methods. We must recognize, however, that this is not a summarization task, as evaluation of a highlighting model must be cruicially different: Summarization tasks produce summaries that are evaluated as relevant, but separate from the document whereas the evaluation of highlight mod- els must be done within the direct context of the doc- ument. Though the approach may be similar, the evaluation is fundamentally different and likely in- fluences the structure of the model applied. In order to confirm this is both useful and possi- ble, however, we must first answer two questions: 1) Can humans agree on which sentences in a docu- ment should be highlighted? Knowing whether hu- man readers distinguish a specific set of sentences from a document will inform to what extent this task differs from extractive summarization tasks, as it would identify how human highlights and baseline summarization model “highlights” differ, if at all. 2) How do current summarization models, trained for their original application, perform when used out- of-the-box as highlight generators? To answer these two research questions, we cre- ated a novel method of collecting data from hu- man annotators via Amazon Mechanical Turk or any other crowdsourcing platform. This paper …

Content truncated to 500 words