Segmenting Subtitles for Correcting ASR Segmentation Errors

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 2842–2854 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 2842 Segmenting Subtitles for Correcting ASR Segmentation Errors David Wan1, Chris Kedzie1, Faisal Ladhak1, Elsbeth Turcan1, Petra Galuˇsˇc´akov´a2, Elena Zotkina2, Zhengping Jiang1, Peter Bell 3, Kathleen McKeown1 1 Columbia University, 2 University of Maryland, 3 University of Edinburgh {dw2735, zj2265}@columbia.edu, {petra, elena}@umiacs.umd.edu, {kedzie, faisal, eturcan, kathy}@cs.columbia.edu, peter.bell@ed.ac.uk Abstract Typical ASR systems segment the input audio into utterances using purely acoustic informa- tion, which may not resemble the sentence- like units that are expected by conventional machine translation (MT) systems for Spoken Language Translation. In this work, we pro- pose a model for correcting the acoustic seg- mentation of ASR models for low-resource languages to improve performance on down- stream tasks. We propose the use of subtitles as a proxy dataset for correcting ASR acous- tic segmentation, creating synthetic acoustic utterances by modeling common error modes. We train a neural tagging model for correct- ing ASR acoustic segmentation and show that it improves downstream performance on MT and audio-document cross-language informa- tion retrieval (CLIR). 1 Introduction Typical ASR systems segment the input audio into utterances using purely acoustic information, i.e., pauses in speaking or other dips in the audio signal, which may not resemble the sentence-like units that are expected by conventional MT systems for spo- ken language translation (SLT) (Cho et al., 2017). Longer utterances may span multiple sentences, while shorter utterances may be sentence fragments containing only a few words (see Figure 1 for exam- ples). Both can be problematic for downstream MT systems. In this work, we propose a model for cor- recting the acoustic segmentation of an ASR model to improve performance on downstream tasks, fo- cusing on the challenges inherent to SLT pipelines for low-resource languages. While prior work has trained intermediate com- ponents to segment ASR output into sentence-like units (Matusov et al., 2007; Rao et al., 2007), these have primarily focused on highly resourced lan- guage pairs such as Arabic and Chinese. When the source language is low-resource, suitable train- ing data may be very limited for ASR and MT, and even nonexistent for segmentation. Since typ- ical low-resource language ASR datasets crawled from the web do not have hand-annotated segments, we propose deriving proxy segmentation datasets from film and television subtitles. Subtitles typi- cally contain segment boundary information like sentence-final punctuation, and while they are not exact transcriptions, they are closer to transcribed speech than many other large text corpora. Our proposed model takes as input a sequence of tokens and segmentation boundaries produced by the acoustic segmentation of the ASR system and returns a corrected segmentation. While subtitles are often similar to speech transcripts, they lack an existing acoustic segmentation for our model to correct. To account for this, we generate synthetic acoustic segmentation by explicitly modeling two common error modes of ASR acoustic segmenta- tion: under- and over-segmentation. We evaluate the downstream MT performance …

Content truncated to 500 words