Text Segmentation by Cross Segment Attention

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 4707–4716, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 4707 Text Segmentation by Cross Segment Attention Michal Lukasik, Boris Dadachev, Kishore Papineni, Gonc¸alo Sim˜oes Google Research {mlukasik,bdadachev,papineni,gsimoes}@google.com Abstract Document and discourse segmentation are two fundamental NLP tasks pertaining to breaking up text into constituents, which are commonly used to help downstream tasks such as infor- mation retrieval or text summarization. In this work, we propose three transformer-based ar- chitectures and provide comprehensive com- parisons with previously proposed approaches on three standard datasets. We establish a new state-of-the-art, reducing in particular the er- ror rates by a large margin in all cases. We further analyze model sizes and find that we can build models with many fewer parameters while keeping good performance, thus facili- tating real-world applications. 1 Introduction Text segmentation is a traditional NLP task that breaks up text into constituents, according to prede- fined requirements. It can be applied to documents, in which case the objective is to create logically coherent sub-document units. These units, or seg- ments, can be any structure of interest, such as paragraphs or sections. This task is often referred to as document segmentation or sometimes simply text segmentation. In Figure 1 we show one ex- ample of document segmentation from Wikipedia, on which the task is typically evaluated (Koshorek et al., 2018; Badjatiya et al., 2018). Documents are often multi-modal, in that they cover multiple aspects and topics; breaking a doc- ument into uni-modal segments can help improve and/or speed up down stream applications. For example, document segmentation has been shown to improve information retrieval by indexing sub- document units instead of full documents (Llopis et al., 2002; Shtekh et al., 2018). Other applications such as summarization and information extraction can also benefit from text segmentation (Koshorek et al., 2018). Early life and marriage: Franklin Delano Roosevelt was born on January 30, 1882, in the Hudson Valley town of Hyde Park, New York, to businessman James Roosevelt I and his second wife, Sara Ann Delano. (...) Aides began to refer to her at the time as “the president’s girl- friend”, and gossip linking the two romantically appeared in the newspapers. (...) Legacy: Roosevelt is widely considered to be one of the most important figures in the history of the United States, as well as one of the most influential figures of the 20th century. (...) Roosevelt has also appeared on several U.S. Postage stamps. Figure 1: Illustration of text segmentation on the ex- ample of the Wikipedia page of President Roosevelt. The aim of document segmentation is breaking the raw text into a sequence of logically coherent sections (e.g., “Early life and marriage” and “Legacy” in our exam- ple). A related task called discourse segmentation breaks up pieces of text into sub-sentence elements called Elementary Discourse Units (EDUs). EDUs are the minimal units in discourse analysis accord- ing to the Rhetorical Structure Theory (Mann and Thompson, 1988). In Figure 2 we …

Content truncated to 500 words