Improving Context Modeling in Neural Topic Segmentation

Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, pages 626–636 December 4 - 7, 2020. c⃝2020 Association for Computational Linguistics 626 Improving Context Modeling in Neural Topic Segmentation Linzi Xing†, Brad Hackinen‡, Giuseppe Carenini†, Francesco Trebbi§ † University of British Columbia, Vancouver, Canada ‡ Ivey Business School, London, Canada § University of California Berkeley, California, USA {lzxing, carenini}@cs.ubc.ca bhackinen@ivey.ca ftrebbi@berkeley.edu Abstract Topic segmentation is critical in key NLP tasks and recent works favor highly effective neu- ral supervised approaches. However, current neural solutions are arguably limited in how they model context. In this paper, we enhance a segmenter based on a hierarchical attention BiLSTM network to better model context, by adding a coherence-related auxiliary task and restricted self-attention. Our optimized seg- menter1 outperforms SOTA approaches when trained and tested on three datasets. We also the robustness of our proposed model in do- main transfer setting by training a model on a large-scale dataset and testing it on four chal- lenging real-world benchmarks. Furthermore, we apply our proposed strategy to two other languages (German and Chinese), and show its effectiveness in multilingual scenarios. 1 Introduction Topic segmentation is a fundamental NLP task that has received considerable attention in recent years (Barrow et al., 2020; Glavas and Somasundaran, 2020; Lukasik et al., 2020). It can reveal impor- tant aspects of a document semantic structure by splitting the document into topical-coherent tex- tual units. Taking the Wikipedia article in Table 1 as an example, without the section marks, a reli- able topic segmenter should be able to detect the correct boundaries within the text and chunk this article into the topical-coherent units T1, T2 and T3. The results of topic segmentation can further benefit other key downstream NLP tasks such as document summarization (Mitra et al., 1997; Riedl and Biemann, 2012a; Xiao and Carenini, 2019), question answering (Oh et al., 2007; Diefenbach et al., 2018), machine reading (van Dijk, 1981; 1Our code will be publicly available at www.cs. ubc.ca/cs-research/lci/research-groups/ natural-language-processing/ Preface: Marcus is a city in Cherokee County, Iowa, United States. [T1] History: S1: The first building in Marcus was erected in 1871. S2: Marcus was incorporated on May 15, 1882. [T2] Geography: S3: Marcus is located at (42.822892, -95.804894). S4: According to the United States Census Bureau, the city has a total area of 1.54 square miles, all land. [T3] Demographics: S5: As of the census of 2010, there were 1,117 people, 494 households, and 310 families residing in the city. ... ... Table 1: A Wikipedia sample article about City Marcus covering three topics: T1, T2 and T3 Saha et al., 2019) and dialogue modeling (Xu et al., 2020; Zhang et al., 2020). A wide variety of techniques have been proposed for topic segmentation. Early unsupervised mod- els exploit word statistic overlaps (Hearst, 1997; Galley et al., 2003), Bayesian contexts (Eisenstein and Barzilay, 2008) or semantic relatedness graphs (Glavaˇs et al., 2016) to measure …

Content truncated to 500 words