Curricular Next Conversation Prediction Pretraining for Transcript Segmentation

Findings of the Association for Computational Linguistics: EACL 2023, pages 2597–2607 May 2-6, 2023 ©2023 Association for Computational Linguistics Curricular Next Conversation Prediction Pretraining for Transcript Segmentation Anvesh Rao Vijjini1* Hanieh Deilamsalehy2 Franck Dernoncourt2 Snigdha Chaturvedi1 1UNC Chapel Hill {anvesh,snigdha}@cs.unc.edu 2Adobe Research {deilamsa,franck.dernoncourt}@adobe.com Abstract Transcript segmentation is the task of divid- ing a single continuous transcript into multiple segments. While document segmentation is a popular task, transcript segmentation has sig- nificant challenges due to the relatively noisy and sporadic nature of data. We propose pre- training strategies to address these challenges. The strategies are based on “Next Conversation Prediction” (NCP) with the underlying idea of pretraining a model to identify consecutive con- versations. We further introduce “Advanced NCP” to make the pretraining task more rele- vant to the downstream task of segmentation break prediction while being significantly eas- ier. Finally we introduce a curriculum to Ad- vanced NCP (Curricular NCP) based on the similarity between pretraining and downstream task samples. Curricular NCP applied to a state- of-the-art model for text segmentation outper- forms prior results. We also show that our pre- training strategies make the model robust to speech recognition errors commonly found in automatically generated transcripts. 1 Introduction Text segmentation is the task of identifying seg- ment breaks to organize a continuous text into se- mantically independent segments. Prior research in text segmentation has largely focused on segment- ing documents such as Wikipedia articles (docu- ment segmentation) (Lukasik et al., 2020; Zhang et al., 2019; Badjatiya et al., 2018; Koshorek et al., 2018) or dialogues such as chat or text messages (dialogue segmentation) (Hsueh et al., 2006; Ar- guello and Rosé, 2006; Xia et al., 2022; Xing and Carenini, 2021). In this paper we address text seg- mentation of transcripts (transcript segmentation). Figure 1 shows examples of segments in transcript data. Transcript segmentation can help summa- rize long videos, podcasts or meetings by segment- ing and summarizing the transcript such as “Video * Work done during internship at Adobe Research. Figure 1: Transcript Segmentation example from the SliceCast-Podcast (Midei and Mandic, 2019) dataset. Here each line indicates start of a new sentence and segment breaks are noted with “[BREAK]”. chapters” in YouTube videos or “Outline Genera- tion” (Zhang et al., 2019) from documents. However, only few works have addressed seg- mentation of transcripts (Midei and Mandic, 2019; Jing et al., 2021; Gruenstein et al., 2008). As shown in Figure 1, transcripts consist of a mix of short sentences, utterances, interjections and long form document style answers. Unlike Wikipedia articles or chat, the sporadic and non uniform flow of text in transcripts makes annotation of segment breaks hard even for humans (Gruenstein et al., 2008). Furthermore, transcripts often involve Automatic Speech Recognition errors such as insertions, dele- tions, replacement and lack of proper punctuation which add to the challenges. As a result of these challenges, most labeled transcript segmentation datasets are small in size making it difficult for models to be trained on them. To address this issue, we propose …

Content truncated to 500 words