Adapting End-to-End Speech Recognition for Readable Subtitles

Proceedings of the 17th International Conference on Spoken Language Translation (IWSLT), pages 247–256 July 9-10, 2020. c⃝2020 Association for Computational Linguistics https://doi.org/10.18653/v1/P17 247 Adapting End-to-End Speech Recognition for Readable Subtitles Danni Liu, Jan Niehues, Gerasimos Spanakis Department of Data Science and Knowledge Engineering, Maastricht University {danni.liu,jan.niehues,jerry.spanakis}@maastrichtuniversity.nl Abstract Automatic speech recognition (ASR) systems are primarily evaluated on transcription accu- racy. However, in some use cases such as sub- titling, verbatim transcription would reduce output readability given limited screen size and reading time. Therefore, this work fo- cuses on ASR with output compression, a task challenging for supervised approaches due to the scarcity of training data. We first inves- tigate a cascaded system, where an unsuper- vised compression model is used to post-edit the transcribed speech. We then compare sev- eral methods of end-to-end speech recognition under output length constraints. The exper- iments show that with limited data far less than needed for training a model from scratch, we can adapt a Transformer-based ASR model to incorporate both transcription and compres- sion capabilities. Furthermore, the best perfor- mance in terms of WER and ROUGE scores is achieved by explicitly modeling the length constraints within the end-to-end ASR system. 1 Introduction Automatic speech recognition (ASR) has become ubiquitous in human interaction with digital de- vices, such as keyboard voice inputs (He et al., 2019) and virtual home assistants (Li et al., 2017). While transcription accuracy is often the primary goal when designing ASR systems, in some use cases the readability of outputs is crucial to user experience. A prominent example is subtitling for TV. In this case, the audience need to multitask, i.e. simultaneously watch video contents, listen to speech utterances, and read subtitles. To avoid a visual overload, not every spoken word needs to be displayed. Meanwhile, the shortened subtitles must still retain the meaning of the spoken content. Moreover, large deviations from the original utter- ance are also undesirable, as the disagreement with auditory input would create a distraction. To compress the subtitles, one straightforward approach is to post-process ASR transcriptions. The task of sentence compression has been well- studied (Knight and Marcu, 2002; Clarke and La- pata, 2006; Rush et al., 2015). In extractive com- pression (Filippova et al., 2015; Angerbauer et al., 2019), only deletion operations are performed on the input. Despite the simplicity, this approach tends to produce outputs that are less grammati- cal (Knight and Marcu, 2002). On the other hand, abstractive compression (Cohn and Lapata, 2008; Rush et al., 2015; Chopra et al., 2016; Yu et al., 2018) involves more sophisticated input reformu- lation, such as word reordering and paraphrasing (Clarke and Lapata, 2006). For the task of com- pressing subtitles, however, the extent of rewriting must be controlled in order to retain consistency with spoken utterances. From a practical point of view, building a sen- tence compression system typically requires train- ing corpora where the target sequences are sum- marized. For most languages and domains, there exists scarcely any resource suitable for supervised training. This …

Content truncated to 500 words