Improved Open Source Automatic Subtitling for Lecture Videos

Improved Open Source Automatic Subtitling for Lecture Videos Robert Geislinger1,2 Benjamin Milde1,2 Chris Biemann1 1Language Technology Group, Universit¨at Hamburg, Germany 2Hamburger Informatik Technologie-Center e.V., Germany robert.geislinger@uni-hamburg.de benjamin.milde@uni-hamburg.de christian.biemann@uni-hamburg.de Abstract This paper summarizes the current state of de- velopment in improving an open source subti- tling tool. This includes improvements to the speech recognition model for German, the re- placement for the punctuation reconstruction architecture and the addition of an audio seg- mentation. The goal of these adjustments is an overall better subtitle quality. The most crucial part of the existing pipeline, the Ger- man speech recognition, is replaced by a new Kaldi TDNN-HMM model trained on 70% of additional audio data, resulting in a word er- ror rate of 6.9% on Tuda-De. The punctuation reconstruction model for German texts is re- placed by a Transformer-based approach that is also trained on new data. English is added as a fully supported second language, including speech recognition and punctuation reconstruc- tion models. Furthermore, to improve speech recognition in long videos, audio segmentation was also added into the pipeline to support long videos flawlessly without quality issues. 1 Introduction Remote learning with lecture videos has become the norm in the Covid-19 pandemic. Subtitling videos make them accessible for persons with hear- ing limitations. Since subtitling videos by hand is a time-consuming and cost-intensive task, this work offers a solution for automatic subtitling. Au- tomatic speech recognition (ASR) is the most im- portant step in the creation of subtitles, but for suf- ficient results, the text must also be supplemented with punctuation marks and be separated at appro- priate places to achieve a good reading flow. This paper presents the results of a revised pipeline to create German and English subtitles with open source algorithms and models. It also introduces the addition of audio segmentation as well as improvements to automatic speech recog- nition and punctuation reconstruction models. The entire pipeline is shown in Figure 1. The model for German ASR was revised and a model for English language was added. Also, the existing punctu- ation reconstruction model is replaced by a new Transformer-based architecture and trained on new data. It is now also possible to get live status in- formation about the current processing step via a Redis database. The tool is already in operation at the Universit¨at Hamburg lecture video portal Lecture2Go1 and the generated subtitles serve as a starting point for fur- ther manual annotation. Users of the platform can also correct the subtitles with a web-based subtitle editor. 2 Related Work Generating subtitles with ASR can be performed both semi-automatically and automatically. In semi-automatic generation systems, texts are re- spoken in a controlled environment by a trained speaker (Sperber et al., 2013; Romero-Fresco, 2020; Vashistha et al., 2017). However, automatic systems are already being used to subtitle videos and conferences (Milde et al., 2021; Geislinger et al., 2021). There are several models for German speech recognition available. A model based on Kaldi TDNN-HMM with ARPA rescoring and RNNLM achieved a …

Content truncated to 500 words