The NiuTrans’s Submission to the IWSLT22 English-to-Chinese Offline Speech Translation Task

Proceedings of the 19th International Conference on Spoken Language Translation (IWSLT 2022), pages 1 - 10 May 26-27, 2022 c⃝2022 Association for Computational Linguistics SubER: A Metric for Automatic Evaluation of Subtitle Quality Patrick Wilken AppTek Aachen, Germany pwilken@apptek.com Panayota Georgakopoulou Athena Consultancy Athens, Greece yota@athenaconsultancy.eu Evgeny Matusov AppTek Aachen, Germany ematusov@apptek.com Abstract This paper addresses the problem of evaluat- ing the quality of automatically generated sub- titles, which includes not only the quality of the machine-transcribed or translated speech, but also the quality of line segmentation and subtitle timing. We propose SubER - a single novel metric based on edit distance with shifts that takes all of these subtitle properties into account. We compare it to existing metrics for evaluating transcription, translation, and sub- title quality. A careful human evaluation in a post-editing scenario shows that the new met- ric has a high correlation with the post-editing effort and direct human assessment scores, out- performing baseline metrics considering only the subtitle text, such as WER and BLEU, and existing methods to integrate segmentation and timing features. 1 Introduction The use of automatically created subtitles has be- come popular due to improved speech recognition (ASR) and machine translation (MT) quality in re- cent years. Most notably, they are used on the web to make content available to a broad audience in a cost-efficient and scalable way. They also gain at- traction in the media industry, where they can be an aid to professional subtitlers and lead to increased productivity. In this work, we address the problem of measur- ing the quality of such automatic subtitling systems. We argue that existing metrics which compare the plain text output of an ASR or MT system to a reference text are not sufficient to reflect the par- ticularities of the subtitling task. We consider two use cases: 1) running speech recognition on the audio track of a video to create subtitles in the orig- inal language; 2) translating existing subtitle files with an MT system. For the first case, the word error rate (WER) of the ASR system is a natural choice for quality control. For MT there exist a wider range of automatic metrics such as BLEU (Papineni et al., 2002), TER (Snover et al., 2006), chrF (Popovi´c, 2015) and, more recently, learned metrics like BertScore (Zhang et al., 2019) and COMET (Rei et al., 2020). These existing metrics are suited to measure the quality of ASR and MT in terms of recognized or translated content only. However, subtitles are de- fined by more than just their textual content: they include timing information, as well as formatting with possible line breaks within a sentence in syn- tactically and semantically proper positions. Figure 1 shows examples of subtitle files in the common SubRip text (SRT) format. Evidently, it differs from plain text, in particular: • The text is segmented into blocks. These blocks are distinct from sentences. A sentence can span several blocks, a block can contain multiple sentences. • A block may …

Content truncated to 500 words