SECTOR: A Neural Model for Coherent Topic Segmentation and Classification

SECTOR: A Neural Model for Coherent Topic Segmentation and Classification Sebastian Arnold Rudolf Schneider Beuth University of Applied Sciences Berlin, Germany {sarnold, ruschneider}@ beuth-hochschule.de Philippe Cudr´e-Mauroux University of Fribourg Fribourg, Switzerland pcm@unifr.ch Felix A. Gers Alexander L¨oser Beuth University of Applied Sciences Berlin, Germany {gers, aloeser}@ beuth-hochschule.de Abstract When searching for information, a human reader first glances over a document, spots relevant sections, and then focuses on a few sentences for resolving her intention. How- ever, the high variance of document structure complicates the identification of the salient topic of a given section at a glance. To tackle this challenge, we present SECTOR, a model to support machine reading systems by segmenting documents into coherent sections and assigning topic labels to each section. Our deep neural network architecture learns a latent topic embedding over the course of a document. This can be leveraged to classify local topics from plain text and segment a document at topic shifts. In addition, we contribute WikiSection, a publicly available data set with 242k labeled sections in English and German from two distinct domains: dis- eases and cities. From our extensive evaluation of 20 architectures, we report a highest score of 71.6% F1 for the segmentation and classification of 30 topics from the English city domain, scored by our SECTOR long short-term memory model with Bloom filter embeddings and bidirectional segmentation. This is a significant improvement of 29.5 points F1 over state-of-the-art CNN classifiers with baseline segmentation. 1 Introduction Today’s systems for natural language understand- ing are composed of building blocks that extract semantic information from the text, such as named entities, relations, topics, or discourse structure. In traditional natural language processing (NLP), these extractors are typically applied to bags of words or full sentences (Hirschberg and Manning, 2015). Recent neural architectures build upon pre- trained word or sentence embeddings (Mikolov et al., 2013; Le and Mikolov, 2014), which focus on semantic relations that can be learned from large sets of paradigmatic examples, even from long ranges (Dieng et al., 2017). From a human perspective, however, it is mostly the authors themselves who help best to understand a text. Especially in long documents, an author thoughtfully designs a readable structure and guides the reader through the text by arranging topics into coherent passages (Glavaˇs et al., 2016). In many cases, this structure is not formally ex- pressed as section headings (e.g., in news articles, reviews, discussion forums) or it is structured according to domain-specific aspects (e.g., health reports, research papers, insurance documents). Ideally, systems for text analytics, such as topic detection and tracking (TDT) (Allan, 2002), text summarization (Huang et al., 2003), information retrieval (IR) (Dias et al., 2007), or question an- swering (QA) (Cohen et al., 2018), could access a document representation that is aware of both topical (i.e., latent semantic content) and struc- tural information (i.e., segmentation) in the text (MacAvaney et al., 2018). The challenge in building such a representation is to combine these two dimensions that are strongly interwoven in the …

Content truncated to 500 words