Topical Segmentation: a Study of Human Performance and a New Measure of Quality.

2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 211–220, Montr´eal, Canada, June 3-8, 2012. c⃝2012 Association for Computational Linguistics Topical Segmentation: a Study of Human Performance and a New Measure of Quality Anna Kazantseva School of Electrical Engineering and Computer Science, University of Ottawa ankazant@eecs.uottawa.ca Stan Szpakowicz School of Electrical Engineering and Computer Science, University of Ottawa & Institute of Computer Science, Polish Academy of Sciences szpak@eecs.uottawa.ca Abstract In a large-scale study of how people find top- ical shifts in written text, 27 annotators were asked to mark topically continuous segments in 20 chapters of a novel. We analyze the re- sulting corpus for inter-annotator agreement and examine disagreement patterns. The re- sults suggest that, while the overall agree- ment is relatively low, the annotators show high agreement on a subset of topical breaks – places where most prominent topic shifts occur. We recommend taking into account the prominence of topical shifts when evalu- ating topical segmentation, effectively penal- izing more severely the errors on more impor- tant breaks. We propose to account for this in a simple modification of the windowDiff metric. We discuss the experimental results of evaluat- ing several topical segmenters with and with- out considering the importance of the individ- ual breaks, and emphasize the more insightful nature of the latter analysis. 1 Introduction Topical segmentation is a useful intermediate step in many high-level NLP applications such as in- formation retrieval, automatic summarization and question answering. It is often necessary to split a long document into topically continuous segments. Segmentation may be particularly beneficial when working with documents without overt structure: speech transcripts (Malioutov and Barzilay, 2006), newswire (Misra et al., 2011) or novels (Kazantseva and Szpakowicz, 2011). The customary approach is to cast text segmentation as a binary problem: is there a shift of topic between any two adjacent tex- tual units (e.g., sentences or paragraphs)? While necessary, this simplification is quite crude. Topic in discourse usually changes continually; some shifts are subtle, others – more prominent. The evaluation of text segmentation remains an open research problem. It is a tradition to compile a gold-standard segmentation reference using one or more annotations created by humans. If an auto- matic segmenter agrees with the reference, it is re- warded, otherwise it is penalized (see Section 4 for details). The nature of the task, however, is such that creating and applying a reference segmentation is far from trivial. The identification of topical shifts re- quires discretization of a continuous concept – how much the topic changes between two adjacent units. That is why annotators often operate at different lev- els of granularity. Some people mark only the most prominent topic fluctuations, while others also in- clude finer changes. The task is also necessarily under-defined. In addition to topic changes per se, annotators effectively must classify some rhetorical and pragmatic phenomena – exactly how much it is depends on the document genre. For simplicity we do not directly address …

Content truncated to 500 words